Oct 30 08:53:30.136867 ip-10-0-135-148 systemd[1]: Starting Kubernetes Kubelet... Oct 30 08:53:30.563810 ip-10-0-135-148 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:53:30.563810 ip-10-0-135-148 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 08:53:30.563810 ip-10-0-135-148 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:53:30.563810 ip-10-0-135-148 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:53:30.567247 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567119 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:30.567247 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567248 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567252 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567256 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567259 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567261 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567264 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567266 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567270 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567272 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567275 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567277 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567280 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567282 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567285 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567293 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567297 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567300 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567302 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567305 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567308 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:30.567320 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567313 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567316 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567319 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567322 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567325 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567328 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567331 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567334 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567336 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567339 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567343 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567347 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567349 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567352 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567354 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567357 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567359 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567362 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567364 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:30.567813 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567367 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567369 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567372 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567374 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567376 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567380 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567383 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567386 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567389 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567392 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567395 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567397 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567400 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567402 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567405 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567407 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567410 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567412 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567415 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567417 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:30.568265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567419 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567422 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567424 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567427 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567429 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567432 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567434 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567437 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567439 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567442 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567444 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567447 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567449 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567452 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567454 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567456 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567459 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567461 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567464 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:30.568768 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567467 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567470 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567472 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567475 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567477 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567480 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567482 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567485 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567487 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567489 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567492 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567494 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567497 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567499 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567501 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567504 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567506 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567510 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567514 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:30.569246 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567517 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567520 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567523 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567525 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567527 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567530 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567532 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.567534 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569118 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569126 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569130 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569132 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569135 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569138 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569141 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569145 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569148 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569151 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569153 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569156 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:30.569720 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569158 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569161 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569164 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569166 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569169 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569171 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569174 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569176 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569178 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569181 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569183 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569186 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569188 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569191 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569193 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569196 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569199 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569201 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569203 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:30.570243 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569207 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569211 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569214 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569217 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569219 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569222 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569224 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569227 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569230 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569233 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569235 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569238 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569240 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569243 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569245 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569247 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569250 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569253 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569256 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:30.570723 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569259 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569263 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569267 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569269 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569272 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569275 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569277 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569280 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569282 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569285 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569288 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569290 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569293 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569295 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569298 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569301 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569303 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569306 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569308 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:30.571182 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569311 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569313 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569316 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569318 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569321 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569324 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569326 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569329 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569331 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569333 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569336 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569338 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569341 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569343 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569346 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569348 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569351 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569353 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569356 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569358 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:30.571687 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569361 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569363 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569366 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569368 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569371 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569373 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569376 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569378 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569381 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569384 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569387 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569390 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569392 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569395 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569397 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569400 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569402 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569915 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569921 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569925 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:30.572146 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569927 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569930 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569933 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569936 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569938 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569941 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569943 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569946 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569948 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569951 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569953 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569956 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569958 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569961 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569964 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569966 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569969 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569971 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569973 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569976 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:30.572653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569978 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569980 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569983 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569986 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569988 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569991 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569994 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569996 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.569998 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570001 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570004 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570006 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570008 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570011 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570013 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570016 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570018 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570021 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570023 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570026 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:30.573120 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570028 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570031 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570033 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570035 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570038 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570040 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570044 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570048 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570051 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570054 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570057 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570060 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570064 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570067 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570069 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570072 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570074 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570078 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570082 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:30.573620 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570086 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570088 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570091 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570094 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570097 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570100 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570102 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570105 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570107 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570110 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570113 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570115 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570118 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570121 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570124 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570126 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570129 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570131 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570134 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:30.574065 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570136 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570139 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570141 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570143 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570146 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570149 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570151 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570154 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570156 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570159 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570161 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570164 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570166 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570169 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570171 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570174 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570176 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570179 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570182 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:30.574528 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570184 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570187 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570190 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570193 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570196 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.570198 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572096 2578 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572106 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572112 2578 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572116 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572121 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572124 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572128 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572133 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572136 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572139 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572143 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572146 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572149 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572152 2578 flags.go:64] FLAG: --cgroup-root="" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572155 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572159 2578 flags.go:64] FLAG: --client-ca-file="" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572162 2578 flags.go:64] FLAG: --cloud-provider="external" Oct 30 08:53:30.575010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572165 2578 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572170 2578 flags.go:64] FLAG: --cluster-domain="" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572173 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572176 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572178 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572182 2578 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572186 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572189 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572192 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572195 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572199 2578 flags.go:64] FLAG: --contention-profiling="false" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572201 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572204 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572207 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572210 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572214 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572217 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572220 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572223 2578 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572226 2578 flags.go:64] FLAG: --enable-server="true" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572228 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572232 2578 flags.go:64] FLAG: --event-burst="100" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572235 2578 flags.go:64] FLAG: --event-qps="50" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572238 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 08:53:30.575538 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572241 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572244 2578 flags.go:64] FLAG: --eviction-hard="" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572248 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572251 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572253 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572256 2578 flags.go:64] FLAG: --eviction-soft="" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572259 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572262 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572265 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572268 2578 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572271 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572273 2578 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572276 2578 flags.go:64] FLAG: --feature-gates="" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572280 2578 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572283 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572286 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572289 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572292 2578 flags.go:64] FLAG: --healthz-port="10248" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572296 2578 flags.go:64] FLAG: --help="false" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572299 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-135-148.ec2.internal" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572302 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572305 2578 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572308 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572311 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 08:53:30.576176 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572314 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572318 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572320 2578 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572323 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572325 2578 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572328 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572331 2578 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572334 2578 flags.go:64] FLAG: --kube-reserved="" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572337 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572339 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572342 2578 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572345 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572348 2578 flags.go:64] FLAG: --lock-file="" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572350 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572353 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572356 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572361 2578 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572364 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572367 2578 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572369 2578 flags.go:64] FLAG: --logging-format="text" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572372 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572375 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572378 2578 flags.go:64] FLAG: --manifest-url="" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572380 2578 flags.go:64] FLAG: --manifest-url-header="" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572385 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 08:53:30.576761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572388 2578 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572392 2578 flags.go:64] FLAG: --max-pods="110" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572395 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572398 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572401 2578 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572404 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572406 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572411 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572414 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572421 2578 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572424 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572427 2578 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572430 2578 flags.go:64] FLAG: --pod-cidr="" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572433 2578 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572435 2578 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572438 2578 flags.go:64] FLAG: --pods-per-core="0" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572441 2578 flags.go:64] FLAG: --port="10250" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572443 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572446 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01fa2920a76d3fbe5" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572450 2578 flags.go:64] FLAG: --qos-reserved="" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572453 2578 flags.go:64] FLAG: --read-only-port="10255" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572455 2578 flags.go:64] FLAG: --register-node="true" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572458 2578 flags.go:64] FLAG: --register-with-taints="" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572461 2578 flags.go:64] FLAG: --registry-burst="10" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572464 2578 flags.go:64] FLAG: --registry-qps="5" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572467 2578 flags.go:64] FLAG: --reserved-cpus="" Oct 30 08:53:30.577346 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572469 2578 flags.go:64] FLAG: --reserved-memory="" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572473 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572476 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572479 2578 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572482 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572484 2578 flags.go:64] FLAG: --runonce="false" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572488 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572491 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572494 2578 flags.go:64] FLAG: --seccomp-default="false" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572497 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572500 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572503 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572506 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572510 2578 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572513 2578 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572515 2578 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572518 2578 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572521 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572524 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572526 2578 flags.go:64] FLAG: --system-cgroups="" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572529 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572535 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572537 2578 flags.go:64] FLAG: --tls-cert-file="" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572540 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572544 2578 flags.go:64] FLAG: --tls-min-version="" Oct 30 08:53:30.577989 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572546 2578 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572549 2578 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572552 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572554 2578 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572557 2578 flags.go:64] FLAG: --v="2" Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572561 2578 flags.go:64] FLAG: --version="false" Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572565 2578 flags.go:64] FLAG: --vmodule="" Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572569 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.572572 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572711 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572716 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572719 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572722 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572724 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572727 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572730 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572733 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572735 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572738 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572740 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572744 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:30.578730 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572748 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572752 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572755 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572758 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572761 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572764 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572766 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572769 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572771 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572774 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572776 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572779 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572781 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572784 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572787 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572789 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572791 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572794 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572796 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:30.579258 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572799 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572801 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572805 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572808 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572810 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572813 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572815 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572818 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572822 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572825 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572828 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572831 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572835 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572837 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572840 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572842 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572845 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572847 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572850 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:30.579736 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572853 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572855 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572857 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572860 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572862 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572865 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572867 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572870 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572872 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572875 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572877 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572880 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572882 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572885 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572888 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572891 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572894 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572896 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572899 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572901 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:30.581117 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572903 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572906 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572908 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572914 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572917 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572921 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572924 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572926 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572929 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572931 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572934 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572936 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572939 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572941 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572944 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572946 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572948 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572951 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572953 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572956 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:30.581653 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572958 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572961 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572963 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572966 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572968 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572970 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572973 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572977 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572980 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572983 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572985 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572988 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572990 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572992 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572995 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.572998 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.573024 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:53:30.582137 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573057 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573062 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573065 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573067 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573070 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573072 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573075 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573077 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573080 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573082 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573085 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573087 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573090 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573093 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573095 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573098 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573100 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573102 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573105 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573107 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:30.582538 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573110 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573112 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573114 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573118 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573121 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573123 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573126 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573128 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573131 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573134 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573136 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573139 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573143 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573145 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573148 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573150 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573152 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573155 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573157 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573160 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:30.583045 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573162 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573165 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573167 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573169 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573172 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573174 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573176 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573179 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573182 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573185 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573189 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573192 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573194 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573197 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573200 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573204 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573206 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573209 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573211 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:30.583507 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573214 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573216 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573219 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573221 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573224 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573226 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573229 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573232 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573234 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573237 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573239 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573242 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573244 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573247 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573249 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573251 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573254 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573258 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573261 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:30.583972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573264 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573267 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573270 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573272 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573275 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573277 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573279 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573282 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573284 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573288 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573291 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573293 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573296 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573299 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573301 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573303 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573306 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573308 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573311 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:30.584421 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573313 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573316 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573319 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573322 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573325 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573327 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573330 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573332 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.573334 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.581912 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.581928 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.581942 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.581947 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582021 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582027 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582030 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582034 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582037 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582040 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582043 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:30.585094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582045 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582048 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582050 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582053 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582055 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582057 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582060 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582063 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582065 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582067 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582070 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582072 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582075 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582077 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582080 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582082 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582085 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582087 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582089 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:30.585836 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582092 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582094 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582097 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582107 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582110 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582113 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582115 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582118 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582120 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582123 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582126 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582132 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582136 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582139 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582142 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582144 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582147 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582150 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582152 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:30.586280 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582155 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582157 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582161 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582163 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582166 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582168 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582171 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582173 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582176 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582178 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582181 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582183 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582186 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582188 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582190 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582193 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582196 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582203 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582206 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582208 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:30.586750 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582211 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582213 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582216 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582218 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582220 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582222 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582225 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582227 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582230 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582232 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582234 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582237 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582239 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582242 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582244 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582247 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582249 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582251 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582254 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:30.587265 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582256 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582259 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582261 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582264 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582266 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582268 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582271 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582273 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582276 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582278 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582280 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582288 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582290 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582293 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582295 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582297 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582300 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582302 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582304 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582307 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:30.587732 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582309 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:30.588462 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582312 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:30.588462 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.582337 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:53:30.588462 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582572 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:30.588462 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582578 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:30.588462 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582581 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:30.588462 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582601 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:30.588462 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582605 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:30.588462 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582607 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:30.588462 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582610 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:30.588462 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582612 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:30.588462 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582615 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:30.588462 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582617 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:30.588462 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582627 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:30.588462 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582630 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:30.588462 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582632 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582635 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582638 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582640 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582643 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582645 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582647 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582650 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582652 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582661 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582664 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582666 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582670 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582674 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582677 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582680 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582682 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582684 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:30.589013 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582687 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582689 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582691 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582694 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582696 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582698 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582701 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582704 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582706 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582708 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582711 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582713 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582716 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582718 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582721 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582723 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582725 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582728 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582730 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582732 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:30.589447 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582735 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582737 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582741 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582744 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582752 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582754 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582757 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582759 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582761 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582764 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582766 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582768 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582771 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582773 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582775 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582778 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582781 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582783 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582785 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:30.590094 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582788 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582790 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582792 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582795 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582797 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582799 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582802 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582804 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582807 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582809 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582818 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582822 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582825 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582827 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582829 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582832 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582834 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582837 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582845 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582848 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:30.590535 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582850 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:30.591101 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582852 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:30.591101 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582855 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:30.591101 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582858 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:30.591101 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582860 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:30.591101 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582862 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:30.591101 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582865 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:30.591101 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582867 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:30.591101 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582870 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:30.591101 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582872 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:30.591101 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582875 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:30.591101 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582877 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:30.591101 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582880 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:30.591101 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582882 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:30.591101 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582885 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:30.591101 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582887 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:30.591101 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582890 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:30.591476 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.582913 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:53:30.591476 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582954 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:30.591476 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582959 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:30.591476 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582962 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:30.591476 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582965 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:30.591476 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582967 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:30.591476 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582970 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:30.591476 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582972 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:30.591476 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582975 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:30.591476 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582977 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:30.591476 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582980 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:30.591476 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582983 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:30.591476 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582985 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:30.591476 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582988 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:30.591476 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582995 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.582998 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583003 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583005 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583008 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583010 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583014 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583018 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583021 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583024 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583027 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583030 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583033 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583035 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583037 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583040 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583042 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583044 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583047 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:30.591962 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583049 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583052 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583055 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583057 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583060 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583062 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583065 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583067 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583069 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583072 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583074 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583076 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583079 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583081 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583089 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583094 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583097 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583100 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583104 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583107 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:30.592404 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583109 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583112 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583114 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583116 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583119 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583121 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583124 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583126 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583129 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583131 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583134 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583136 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583139 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583141 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583143 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583146 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583148 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583150 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583152 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583155 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:30.592999 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583157 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583160 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583162 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583165 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583167 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583172 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583175 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583185 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583188 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583190 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583194 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583196 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583199 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583202 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583204 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583207 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583209 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583212 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583214 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583217 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:30.593832 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583219 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583221 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583224 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583226 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583228 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583231 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583233 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583236 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583238 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583240 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583244 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583246 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583249 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:30.583251 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.583306 2578 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.589642 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.590825 2578 server.go:1014] "Starting client certificate rotation" Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.590927 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 08:53:30.594787 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.590969 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 08:53:30.595432 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.593364 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 08:53:30.595432 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.595193 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 08:53:30.605333 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.605318 2578 log.go:25] "Validated CRI v1 runtime API" Oct 30 08:53:30.609197 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.609181 2578 log.go:25] "Validated CRI v1 image API" Oct 30 08:53:30.609974 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.609951 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 08:53:30.614638 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.614617 2578 fs.go:135] Filesystem UUIDs: map[2f8ac2d6-f89e-402b-b321-a5b8c8b6d6c1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fb73dd3f-f069-419b-b592-f501e3eab477:/dev/nvme0n1p3] Oct 30 08:53:30.614728 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.614636 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 08:53:30.619820 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.619710 2578 manager.go:217] Machine: {Timestamp:2026-10-30 08:53:30.618219083 +0000 UTC m=+0.371370665 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093849 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26215c761113342d327dfe5c750350 SystemUUID:ec26215c-7611-1334-2d32-7dfe5c750350 BootID:8a3299f6-62fb-4bbc-9e80-3158f1cc740b Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e6:33:d6:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e6:33:d6:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:cc:35:c6:bd:33 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 08:53:30.619820 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.619816 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 08:53:30.619944 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.619887 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 08:53:30.622624 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.622573 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 08:53:30.622744 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.622626 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-148.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 08:53:30.622787 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.622746 2578 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 08:53:30.622787 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.622755 2578 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 08:53:30.622787 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.622784 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 08:53:30.622872 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.622801 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 08:53:30.624130 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.624119 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 08:53:30.628723 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.628711 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 08:53:30.628829 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.628820 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 08:53:30.629678 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.629669 2578 kubelet.go:493] "Attempting to sync node with API server" Oct 30 08:53:30.629711 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.629683 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 08:53:30.629711 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.629692 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 08:53:30.629711 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.629702 2578 kubelet.go:405] "Adding apiserver pod source" Oct 30 08:53:30.629711 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.629710 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 08:53:30.630065 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.630050 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 08:53:30.634395 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.634368 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 08:53:30.641697 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.641684 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 08:53:30.642676 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.642665 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 08:53:30.642722 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.642695 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 08:53:30.642852 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.642842 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 08:53:30.642883 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.642858 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 08:53:30.642883 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.642864 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 08:53:30.642883 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.642869 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 08:53:30.642883 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.642875 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 08:53:30.642883 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.642880 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 08:53:30.643553 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.643545 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 08:53:30.643596 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.643553 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 08:53:30.643596 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.643560 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 08:53:30.643596 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.643566 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 08:53:30.643596 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.643574 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 08:53:30.643756 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.643599 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 08:53:30.644345 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.644337 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 08:53:30.644376 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.644346 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 08:53:30.648502 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.648489 2578 server.go:1263] "Started kubelet" Oct 30 08:53:30.648659 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.648632 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 08:53:30.648707 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.648617 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 08:53:30.648707 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.648687 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 08:53:30.649178 ip-10-0-135-148 systemd[1]: Started Kubernetes Kubelet. Oct 30 08:53:30.650231 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.650216 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 08:53:30.651492 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.651478 2578 server.go:317] "Adding debug handlers to kubelet server" Oct 30 08:53:30.654907 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:30.654874 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 08:53:30.657452 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.657432 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 08:53:30.658834 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.658819 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 08:53:30.658950 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:30.658921 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found" Oct 30 08:53:30.659696 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.659678 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 08:53:30.659696 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.659696 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 08:53:30.659809 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.659686 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 08:53:30.659809 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.659757 2578 factory.go:55] Registering systemd factory Oct 30 08:53:30.659809 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.659785 2578 factory.go:223] Registration of the systemd container factory successfully Oct 30 08:53:30.659921 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.659811 2578 reconstruct.go:97] "Volume reconstruction finished" Oct 30 08:53:30.659921 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.659818 2578 reconciler.go:29] "Reconciler: start to sync state" Oct 30 08:53:30.660017 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.660000 2578 factory.go:153] Registering CRI-O factory Oct 30 08:53:30.660017 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.660015 2578 factory.go:223] Registration of the crio container factory successfully Oct 30 08:53:30.660149 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.660055 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 08:53:30.660149 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.660069 2578 factory.go:103] Registering Raw factory Oct 30 08:53:30.660149 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.660081 2578 manager.go:1199] Started watching for new ooms in manager Oct 30 08:53:30.661417 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.661403 2578 manager.go:319] Starting recovery of all containers Oct 30 08:53:30.673052 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.673034 2578 manager.go:324] Recovery completed Oct 30 08:53:30.677394 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.677381 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:30.679187 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.679167 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:30.679263 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.679203 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:30.679263 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.679214 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:30.680126 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.680110 2578 cpu_manager.go:226] "Starting" policy="none" Oct 30 08:53:30.680126 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.680123 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 08:53:30.680290 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.680139 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 08:53:30.680290 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.680155 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:30.680818 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.680802 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:30.680881 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.680828 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:30.680881 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.680837 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:30.682328 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.682317 2578 policy_none.go:50] "Start" Oct 30 08:53:30.682362 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.682333 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 08:53:30.682362 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.682343 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 08:53:30.684379 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.684366 2578 policy_none.go:44] "Start" Oct 30 08:53:30.685283 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.685165 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 08:53:30.685324 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.685243 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jqs4l" Oct 30 08:53:30.699434 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.699405 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-148.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 08:53:30.700025 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:30.700003 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 08:53:30.701053 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:30.699801 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-148.ec2.internal.18e342b268809fcd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-148.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-148.ec2.internal,},FirstTimestamp:2026-10-30 08:53:30.648461261 +0000 UTC m=+0.401612843,LastTimestamp:2026-10-30 08:53:30.648461261 +0000 UTC m=+0.401612843,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-148.ec2.internal,}" Oct 30 08:53:30.712255 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:30.712230 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 08:53:30.712351 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:30.712286 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 08:53:30.718200 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.718162 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 08:53:30.721298 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.721284 2578 manager.go:341] "Starting Device Plugin manager" Oct 30 08:53:30.721361 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:30.721314 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 08:53:30.721361 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.721323 2578 server.go:87] "Starting device plugin registration server" Oct 30 08:53:30.721552 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.721540 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 08:53:30.721604 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.721554 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 08:53:30.721667 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.721648 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 08:53:30.721747 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.721734 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 08:53:30.721796 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.721748 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 08:53:30.722165 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:30.722147 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 08:53:30.722250 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:30.722193 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-148.ec2.internal\" not found" Oct 30 08:53:30.746671 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.746652 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:30.800761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.800737 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 08:53:30.802755 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.802000 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 08:53:30.802755 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.802016 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 08:53:30.802755 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.802030 2578 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 08:53:30.802755 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:30.802063 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 08:53:30.814446 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.814403 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:30.821649 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.821636 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:30.823074 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.823059 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:30.823131 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.823090 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:30.823131 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.823101 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:30.823131 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.823130 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-148.ec2.internal" Oct 30 08:53:30.839688 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.839670 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-148.ec2.internal" Oct 30 08:53:30.839736 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:30.839689 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-148.ec2.internal\": node \"ip-10-0-135-148.ec2.internal\" not found" Oct 30 08:53:30.878792 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:30.878771 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found" Oct 30 08:53:30.903764 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.902968 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal"] Oct 30 08:53:30.903823 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.903791 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:30.904651 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.904637 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:30.904716 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.904661 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:30.904716 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.904670 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:30.906839 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.906820 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:30.906948 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.906935 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:53:30.907002 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.906961 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:30.907562 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.907548 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:30.907648 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.907576 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:30.907648 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.907600 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:30.907648 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.907604 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:30.907648 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.907623 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:30.907648 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.907634 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:30.909709 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.909697 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" Oct 30 08:53:30.909760 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.909735 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:30.910433 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.910417 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:30.910504 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.910470 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:30.910504 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.910498 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:30.942267 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:30.942247 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-148.ec2.internal\" not found" node="ip-10-0-135-148.ec2.internal" Oct 30 08:53:30.944893 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:30.944876 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-148.ec2.internal\" not found" node="ip-10-0-135-148.ec2.internal" Oct 30 08:53:30.961001 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.960982 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f68f752438bee3aab3d894447acc3ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"7f68f752438bee3aab3d894447acc3ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:53:30.961063 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.961008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f68f752438bee3aab3d894447acc3ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"7f68f752438bee3aab3d894447acc3ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:53:30.961063 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:30.961023 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9307784112acd4d13d34c64b57f18e71-config\") pod \"kube-apiserver-proxy-ip-10-0-135-148.ec2.internal\" (UID: \"9307784112acd4d13d34c64b57f18e71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" Oct 30 08:53:30.979277 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:30.979261 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found" Oct 30 08:53:31.061384 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.061361 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f68f752438bee3aab3d894447acc3ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"7f68f752438bee3aab3d894447acc3ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:53:31.061479 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.061390 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f68f752438bee3aab3d894447acc3ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"7f68f752438bee3aab3d894447acc3ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:53:31.061479 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.061402 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f68f752438bee3aab3d894447acc3ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"7f68f752438bee3aab3d894447acc3ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:53:31.061479 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.061415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9307784112acd4d13d34c64b57f18e71-config\") pod \"kube-apiserver-proxy-ip-10-0-135-148.ec2.internal\" (UID: \"9307784112acd4d13d34c64b57f18e71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" Oct 30 08:53:31.061479 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.061442 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f68f752438bee3aab3d894447acc3ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"7f68f752438bee3aab3d894447acc3ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:53:31.061479 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.061445 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9307784112acd4d13d34c64b57f18e71-config\") pod \"kube-apiserver-proxy-ip-10-0-135-148.ec2.internal\" (UID: \"9307784112acd4d13d34c64b57f18e71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" Oct 30 08:53:31.080042 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:31.080001 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found" Oct 30 08:53:31.180948 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:31.180927 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found" Oct 30 08:53:31.245184 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.245157 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:53:31.248260 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.248238 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" Oct 30 08:53:31.281975 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:31.281951 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found" Oct 30 08:53:31.382532 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:31.382481 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found" Oct 30 08:53:31.483029 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:31.483008 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found" Oct 30 08:53:31.583579 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:31.583557 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found" Oct 30 08:53:31.590716 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.590699 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 08:53:31.590843 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.590826 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 08:53:31.590881 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.590852 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 08:53:31.684659 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:31.684630 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found" Oct 30 08:53:31.690832 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.690781 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 08:48:31 +0000 UTC" deadline="2028-06-13 01:37:02.444278558 +0000 UTC" Oct 30 08:53:31.690832 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.690832 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14200h43m30.753452496s" Oct 30 08:53:31.784717 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:31.784694 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found" Oct 30 08:53:31.793872 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:31.793842 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f68f752438bee3aab3d894447acc3ea.slice/crio-f7e3415c41fdab90415e268f8425d6cb2f322f1efec0a924e2e5c8f878c6462f WatchSource:0}: Error finding container f7e3415c41fdab90415e268f8425d6cb2f322f1efec0a924e2e5c8f878c6462f: Status 404 returned error can't find the container with id f7e3415c41fdab90415e268f8425d6cb2f322f1efec0a924e2e5c8f878c6462f Oct 30 08:53:31.794308 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:31.794288 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9307784112acd4d13d34c64b57f18e71.slice/crio-16ed60df4ee79e43d7ef21373cf8cef9830cec68ffb662f192083bbec7dff6fd WatchSource:0}: Error finding container 16ed60df4ee79e43d7ef21373cf8cef9830cec68ffb662f192083bbec7dff6fd: Status 404 returned error can't find the container with id 16ed60df4ee79e43d7ef21373cf8cef9830cec68ffb662f192083bbec7dff6fd Oct 30 08:53:31.799320 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.799303 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 08:53:31.805525 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.805489 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" event={"ID":"7f68f752438bee3aab3d894447acc3ea","Type":"ContainerStarted","Data":"f7e3415c41fdab90415e268f8425d6cb2f322f1efec0a924e2e5c8f878c6462f"} Oct 30 08:53:31.806634 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.806616 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" event={"ID":"9307784112acd4d13d34c64b57f18e71","Type":"ContainerStarted","Data":"16ed60df4ee79e43d7ef21373cf8cef9830cec68ffb662f192083bbec7dff6fd"} Oct 30 08:53:31.854498 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.854477 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:31.855537 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.855523 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:31.855618 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.855549 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:31.855618 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.855560 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:31.884925 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:31.884905 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found" Oct 30 08:53:31.975883 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.975838 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:31.984998 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:31.984984 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found" Oct 30 08:53:31.997981 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:31.997962 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:32.059036 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.059015 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:53:32.081893 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.081874 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:53:32.082781 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.082769 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" Oct 30 08:53:32.099748 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.099729 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:53:32.630675 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.630584 2578 apiserver.go:52] "Watching apiserver" Oct 30 08:53:32.658741 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.658719 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 08:53:32.673995 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.673972 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 08:53:32.674382 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.674356 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-q8z8s","openshift-multus/multus-additional-cni-plugins-hwg67","openshift-multus/network-metrics-daemon-ktt7j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p","openshift-cluster-node-tuning-operator/tuned-nt944","openshift-multus/multus-qgqjn","openshift-network-diagnostics/network-check-target-tf9r8","kube-system/global-pull-secret-syncer-m5sz6","kube-system/konnectivity-agent-hsqk7","openshift-image-registry/node-ca-mnq7z","openshift-network-operator/iptables-alerter-kft2v","openshift-ovn-kubernetes/ovnkube-node-mf5b5"] Oct 30 08:53:32.677248 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.677224 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:32.677363 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:32.677307 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:53:32.679532 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.679508 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:53:32.681993 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.681971 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:32.682106 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:32.682050 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:53:32.684523 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.684496 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.687368 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.687335 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.689927 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.689911 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.692316 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.692296 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:32.692427 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:32.692372 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:53:32.694869 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.694840 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 08:53:32.694869 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.694863 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:53:32.695187 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.695163 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-6v9sh\"" Oct 30 08:53:32.695443 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.695407 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 08:53:32.698534 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.697972 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.699625 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.699583 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 08:53:32.699905 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.699753 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 08:53:32.699905 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.699785 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 08:53:32.699905 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.699861 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 08:53:32.700367 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.700349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:53:32.701633 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.701610 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 08:53:32.701746 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.701652 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wdt5m\"" Oct 30 08:53:32.701809 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.701786 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 08:53:32.704261 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.704114 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.705094 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.705057 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 08:53:32.706583 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.706561 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:53:32.707486 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.707343 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 08:53:32.707486 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.707425 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5ghz4" Oct 30 08:53:32.711061 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.711044 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5ghz4" Oct 30 08:53:32.761029 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.761008 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 08:53:32.772358 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysconfig\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.772358 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772298 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmq2k\" (UniqueName: \"kubernetes.io/projected/f82405fd-bde5-4428-8b23-2be99042feef-kube-api-access-bmq2k\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:32.772358 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-sys-fs\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.772733 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysctl-d\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.772733 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772438 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-socket-dir-parent\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.772733 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772500 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-kubelet\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.772733 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772543 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-hostroot\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.772733 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772607 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-var-lib-kubelet\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.772733 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772646 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-system-cni-dir\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.772733 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysctl-conf\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.773048 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772750 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-ovn\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.773048 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772780 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-etc-kubernetes\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.773048 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772806 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/87f93427-45fe-46a3-8b9f-23d865534d41-iptables-alerter-script\") pod \"iptables-alerter-kft2v\" (UID: \"87f93427-45fe-46a3-8b9f-23d865534d41\") " pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:53:32.773048 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-socket-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.773048 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772869 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7400965-54de-4420-a62a-37baf38d949d-kubelet-config\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:32.773048 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772899 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-log-socket\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.773048 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772924 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-run-ovn-kubernetes\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.773048 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-hosts-file\") pod \"node-resolver-q8z8s\" (UID: \"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6\") " pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:53:32.773048 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.772990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-registration-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.773048 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773032 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx2kb\" (UniqueName: \"kubernetes.io/projected/c032d4ff-047f-43a9-a27e-41affb60615d-kube-api-access-fx2kb\") pod \"node-ca-mnq7z\" (UID: \"c032d4ff-047f-43a9-a27e-41affb60615d\") " pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:53:32.773540 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773068 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-node-log\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.773540 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773104 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovn-node-metrics-cert\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.773540 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-etc-selinux\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.773540 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773220 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-lib-modules\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.773540 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773271 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-systemd\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.773540 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773303 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.773540 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773339 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-systemd\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.773540 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-etc-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.773540 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773424 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-cni-bin\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.773540 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-cnibin\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.773540 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773477 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87f93427-45fe-46a3-8b9f-23d865534d41-host-slash\") pod \"iptables-alerter-kft2v\" (UID: \"87f93427-45fe-46a3-8b9f-23d865534d41\") " pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:53:32.773540 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqch6\" (UniqueName: \"kubernetes.io/projected/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-kube-api-access-fqch6\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.774123 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773566 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-run\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.774123 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773594 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c032d4ff-047f-43a9-a27e-41affb60615d-host\") pod \"node-ca-mnq7z\" (UID: \"c032d4ff-047f-43a9-a27e-41affb60615d\") " pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:53:32.774123 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773685 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b-konnectivity-ca\") pod \"konnectivity-agent-hsqk7\" (UID: \"8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b\") " pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:53:32.774123 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773714 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-cni-bin\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.774123 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773741 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-host\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.774123 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773770 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkdr4\" (UniqueName: \"kubernetes.io/projected/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-kube-api-access-vkdr4\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.774123 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773804 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-slash\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.774123 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773835 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-run-netns\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.774123 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773862 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-var-lib-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.774123 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773913 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.774123 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773963 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-daemon-config\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.774123 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.773996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-tmp-dir\") pod \"node-resolver-q8z8s\" (UID: \"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6\") " pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:53:32.774123 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774038 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-systemd-units\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.774123 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774071 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-netns\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.774123 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774116 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-multus-certs\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.774731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774147 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szrjj\" (UniqueName: \"kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj\") pod \"network-check-target-tf9r8\" (UID: \"25cb63c1-5df3-4be8-b407-b9bf2641cf95\") " pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:32.774731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774180 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56x5w\" (UniqueName: \"kubernetes.io/projected/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-kube-api-access-56x5w\") pod \"node-resolver-q8z8s\" (UID: \"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6\") " pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:53:32.774731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774200 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.774731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-device-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.774731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774245 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.774731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774281 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m49gk\" (UniqueName: \"kubernetes.io/projected/2b94c3af-6055-4f33-bd4b-57d4411d0b28-kube-api-access-m49gk\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.774731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774304 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-cni-multus\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.774731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgsqg\" (UniqueName: \"kubernetes.io/projected/87f93427-45fe-46a3-8b9f-23d865534d41-kube-api-access-kgsqg\") pod \"iptables-alerter-kft2v\" (UID: \"87f93427-45fe-46a3-8b9f-23d865534d41\") " pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:53:32.774731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:32.774731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774389 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-cni-netd\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.774731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774416 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-env-overrides\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.774731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774438 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-conf-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.774731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-tuned\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.774731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774526 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b-agent-certs\") pod \"konnectivity-agent-hsqk7\" (UID: \"8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b\") " pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:53:32.774731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774549 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-kubelet\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.774731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774574 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.775277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-cnibin\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.775277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774622 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-tmp\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.775277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c032d4ff-047f-43a9-a27e-41affb60615d-serviceca\") pod \"node-ca-mnq7z\" (UID: \"c032d4ff-047f-43a9-a27e-41affb60615d\") " pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:53:32.775277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774691 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovnkube-config\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.775277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774706 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovnkube-script-lib\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.775277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774727 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-k8s-cni-cncf-io\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.775277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774742 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psn67\" (UniqueName: \"kubernetes.io/projected/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-kube-api-access-psn67\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.775277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774763 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:32.775277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.775277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774813 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-modprobe-d\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.775277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774836 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-kubernetes\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.775277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774857 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-system-cni-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.775277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774871 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-cni-binary-copy\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.775277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774953 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-cni-binary-copy\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.775277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774980 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgbtw\" (UniqueName: \"kubernetes.io/projected/2eebeab0-d5a7-443d-9ed8-365761816c98-kube-api-access-wgbtw\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.775277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.774999 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-sys\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.775971 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.775014 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7400965-54de-4420-a62a-37baf38d949d-dbus\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:32.775971 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.775028 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-cni-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.805127 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qhn6k\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.805314 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.805901 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.805954 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cnfjl\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806012 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806077 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gbg4x\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806121 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806138 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806204 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806217 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806230 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806254 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806279 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-twrgj\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806303 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806336 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kd468\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806378 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806412 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806447 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806078 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2jshw\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806382 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806505 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806586 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806763 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pw6p6\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806338 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 08:53:32.806844 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.806806 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 08:53:32.828522 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.828500 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875512 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c032d4ff-047f-43a9-a27e-41affb60615d-host\") pod \"node-ca-mnq7z\" (UID: \"c032d4ff-047f-43a9-a27e-41affb60615d\") " pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b-konnectivity-ca\") pod \"konnectivity-agent-hsqk7\" (UID: \"8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b\") " pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-cni-bin\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875621 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-host\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkdr4\" (UniqueName: \"kubernetes.io/projected/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-kube-api-access-vkdr4\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875671 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-slash\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875699 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-run-netns\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875723 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-var-lib-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875761 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-daemon-config\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875826 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-tmp-dir\") pod \"node-resolver-q8z8s\" (UID: \"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6\") " pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-systemd-units\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-netns\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-multus-certs\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875930 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrjj\" (UniqueName: \"kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj\") pod \"network-check-target-tf9r8\" (UID: \"25cb63c1-5df3-4be8-b407-b9bf2641cf95\") " pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56x5w\" (UniqueName: \"kubernetes.io/projected/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-kube-api-access-56x5w\") pod \"node-resolver-q8z8s\" (UID: \"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6\") " pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:53:32.880738 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.875990 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-device-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876067 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m49gk\" (UniqueName: \"kubernetes.io/projected/2b94c3af-6055-4f33-bd4b-57d4411d0b28-kube-api-access-m49gk\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876092 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-cni-multus\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgsqg\" (UniqueName: \"kubernetes.io/projected/87f93427-45fe-46a3-8b9f-23d865534d41-kube-api-access-kgsqg\") pod \"iptables-alerter-kft2v\" (UID: \"87f93427-45fe-46a3-8b9f-23d865534d41\") " pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-cni-netd\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876198 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-env-overrides\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-conf-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-tuned\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b-agent-certs\") pod \"konnectivity-agent-hsqk7\" (UID: \"8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b\") " pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876297 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-kubelet\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-cnibin\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-tmp\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876398 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c032d4ff-047f-43a9-a27e-41affb60615d-serviceca\") pod \"node-ca-mnq7z\" (UID: \"c032d4ff-047f-43a9-a27e-41affb60615d\") " pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:53:32.881542 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovnkube-config\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876449 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovnkube-script-lib\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876475 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-k8s-cni-cncf-io\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876501 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psn67\" (UniqueName: \"kubernetes.io/projected/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-kube-api-access-psn67\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876555 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-modprobe-d\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-kubernetes\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-system-cni-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-cni-binary-copy\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-cni-binary-copy\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbtw\" (UniqueName: \"kubernetes.io/projected/2eebeab0-d5a7-443d-9ed8-365761816c98-kube-api-access-wgbtw\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-sys\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7400965-54de-4420-a62a-37baf38d949d-dbus\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-cni-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysconfig\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmq2k\" (UniqueName: \"kubernetes.io/projected/f82405fd-bde5-4428-8b23-2be99042feef-kube-api-access-bmq2k\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:32.882235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876901 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-sys-fs\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876926 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysctl-d\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876963 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-socket-dir-parent\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.876989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-kubelet\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-hostroot\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-var-lib-kubelet\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-system-cni-dir\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysctl-conf\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877125 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-ovn\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877151 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-etc-kubernetes\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/87f93427-45fe-46a3-8b9f-23d865534d41-iptables-alerter-script\") pod \"iptables-alerter-kft2v\" (UID: \"87f93427-45fe-46a3-8b9f-23d865534d41\") " pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-socket-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877234 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7400965-54de-4420-a62a-37baf38d949d-kubelet-config\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-log-socket\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-run-ovn-kubernetes\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877316 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-hosts-file\") pod \"node-resolver-q8z8s\" (UID: \"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6\") " pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-registration-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.882965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx2kb\" (UniqueName: \"kubernetes.io/projected/c032d4ff-047f-43a9-a27e-41affb60615d-kube-api-access-fx2kb\") pod \"node-ca-mnq7z\" (UID: \"c032d4ff-047f-43a9-a27e-41affb60615d\") " pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-node-log\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovn-node-metrics-cert\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-etc-selinux\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-lib-modules\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877510 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-systemd\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877538 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-systemd\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877591 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-etc-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877617 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-cni-bin\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-cnibin\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877686 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87f93427-45fe-46a3-8b9f-23d865534d41-host-slash\") pod \"iptables-alerter-kft2v\" (UID: \"87f93427-45fe-46a3-8b9f-23d865534d41\") " pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877715 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqch6\" (UniqueName: \"kubernetes.io/projected/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-kube-api-access-fqch6\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877742 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-run\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.877960 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-run\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.878024 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c032d4ff-047f-43a9-a27e-41affb60615d-host\") pod \"node-ca-mnq7z\" (UID: \"c032d4ff-047f-43a9-a27e-41affb60615d\") " pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.878637 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b-konnectivity-ca\") pod \"konnectivity-agent-hsqk7\" (UID: \"8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b\") " pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.878747 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-cni-bin\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.883571 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.878808 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-host\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.879086 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-slash\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.879144 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-run-netns\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.879193 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-var-lib-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.879234 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.879772 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-daemon-config\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.880057 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-tmp-dir\") pod \"node-resolver-q8z8s\" (UID: \"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6\") " pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.880137 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-systemd-units\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.880186 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-netns\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.880232 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-multus-certs\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.880688 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.880762 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-hosts-file\") pod \"node-resolver-q8z8s\" (UID: \"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6\") " pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.880773 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysconfig\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.880772 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-run-ovn-kubernetes\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.880816 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-systemd\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.880828 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-sys-fs\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.880892 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-registration-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.880977 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysctl-d\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.884378 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.881042 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-socket-dir-parent\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.881091 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-kubelet\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.881128 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-node-log\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.881137 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-hostroot\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.881213 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-etc-selinux\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.881310 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-lib-modules\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.881308 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87f93427-45fe-46a3-8b9f-23d865534d41-host-slash\") pod \"iptables-alerter-kft2v\" (UID: \"87f93427-45fe-46a3-8b9f-23d865534d41\") " pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.881374 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-cnibin\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.881431 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-system-cni-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.881579 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-sys\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.881773 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-cni-bin\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.881843 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c032d4ff-047f-43a9-a27e-41affb60615d-serviceca\") pod \"node-ca-mnq7z\" (UID: \"c032d4ff-047f-43a9-a27e-41affb60615d\") " pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.881845 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-etc-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.881932 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-device-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.882046 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.882079 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-etc-kubernetes\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.882126 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-var-lib-kubelet\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.882154 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-system-cni-dir\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.885009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.882239 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysctl-conf\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.882265 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-ovn\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.882292 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-conf-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.882464 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-cni-multus\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.882563 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovnkube-config\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.882638 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.882730 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-cni-netd\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.882733 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7400965-54de-4420-a62a-37baf38d949d-dbus\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.882781 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-cni-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.882992 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovnkube-script-lib\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.883070 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-k8s-cni-cncf-io\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.883199 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-cnibin\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:32.883305 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:32.883361 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs podName:f82405fd-bde5-4428-8b23-2be99042feef nodeName:}" failed. No retries permitted until 2026-10-30 08:53:33.383346873 +0000 UTC m=+3.136498474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs") pod "network-metrics-daemon-ktt7j" (UID: "f82405fd-bde5-4428-8b23-2be99042feef") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.883506 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-cni-binary-copy\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.883518 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-cni-binary-copy\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.883610 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-kubernetes\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.885827 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.883646 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-env-overrides\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.886585 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.883722 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-modprobe-d\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.886585 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.883773 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-kubelet\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.886585 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.883811 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7400965-54de-4420-a62a-37baf38d949d-kubelet-config\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:32.886585 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.883821 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-log-socket\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.886585 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.883867 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.886585 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.883978 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.886585 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.884014 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.886585 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.883983 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-socket-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.886585 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:32.884051 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:32.886585 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:32.884248 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret podName:c7400965-54de-4420-a62a-37baf38d949d nodeName:}" failed. No retries permitted until 2026-10-30 08:53:33.384229444 +0000 UTC m=+3.137381031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret") pod "global-pull-secret-syncer-m5sz6" (UID: "c7400965-54de-4420-a62a-37baf38d949d") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:32.886585 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.882052 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-systemd\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.886585 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.884387 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/87f93427-45fe-46a3-8b9f-23d865534d41-iptables-alerter-script\") pod \"iptables-alerter-kft2v\" (UID: \"87f93427-45fe-46a3-8b9f-23d865534d41\") " pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:53:32.886585 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.886164 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b-agent-certs\") pod \"konnectivity-agent-hsqk7\" (UID: \"8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b\") " pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:53:32.886585 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.886213 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-tmp\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.891924 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.891829 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-tuned\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.892157 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.892034 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovn-node-metrics-cert\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.918768 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:32.918647 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:32.918870 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:32.918843 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:32.918870 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:32.918859 2578 projected.go:196] Error preparing data for projected volume kube-api-access-szrjj for pod openshift-network-diagnostics/network-check-target-tf9r8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:32.918990 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:32.918919 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj podName:25cb63c1-5df3-4be8-b407-b9bf2641cf95 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:33.418900699 +0000 UTC m=+3.172052289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-szrjj" (UniqueName: "kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj") pod "network-check-target-tf9r8" (UID: "25cb63c1-5df3-4be8-b407-b9bf2641cf95") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:32.921418 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.921392 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psn67\" (UniqueName: \"kubernetes.io/projected/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-kube-api-access-psn67\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:53:32.923691 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.923646 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56x5w\" (UniqueName: \"kubernetes.io/projected/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-kube-api-access-56x5w\") pod \"node-resolver-q8z8s\" (UID: \"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6\") " pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:53:32.938090 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.938064 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgbtw\" (UniqueName: \"kubernetes.io/projected/2eebeab0-d5a7-443d-9ed8-365761816c98-kube-api-access-wgbtw\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:32.938319 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.938300 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgsqg\" (UniqueName: \"kubernetes.io/projected/87f93427-45fe-46a3-8b9f-23d865534d41-kube-api-access-kgsqg\") pod \"iptables-alerter-kft2v\" (UID: \"87f93427-45fe-46a3-8b9f-23d865534d41\") " pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:53:32.938423 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.938397 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmq2k\" (UniqueName: \"kubernetes.io/projected/f82405fd-bde5-4428-8b23-2be99042feef-kube-api-access-bmq2k\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:32.938882 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.938858 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkdr4\" (UniqueName: \"kubernetes.io/projected/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-kube-api-access-vkdr4\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:32.938974 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.938945 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqch6\" (UniqueName: \"kubernetes.io/projected/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-kube-api-access-fqch6\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:32.939097 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.939077 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx2kb\" (UniqueName: \"kubernetes.io/projected/c032d4ff-047f-43a9-a27e-41affb60615d-kube-api-access-fx2kb\") pod \"node-ca-mnq7z\" (UID: \"c032d4ff-047f-43a9-a27e-41affb60615d\") " pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:53:32.940741 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.940686 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m49gk\" (UniqueName: \"kubernetes.io/projected/2b94c3af-6055-4f33-bd4b-57d4411d0b28-kube-api-access-m49gk\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:32.995922 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:32.995883 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:53:33.006057 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.006035 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:33.008148 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:33.008126 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4cd17b_9e6f_41d3_970f_9c19d3d1e13b.slice/crio-2dd179b53efad516bbc24bd56ba672e4606abc1638c1eeed9363a97b5cde81a8 WatchSource:0}: Error finding container 2dd179b53efad516bbc24bd56ba672e4606abc1638c1eeed9363a97b5cde81a8: Status 404 returned error can't find the container with id 2dd179b53efad516bbc24bd56ba672e4606abc1638c1eeed9363a97b5cde81a8 Oct 30 08:53:33.022369 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.022349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qgqjn" Oct 30 08:53:33.028328 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.028302 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:33.031164 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.031142 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:53:33.034061 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.034044 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:53:33.034313 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:33.034244 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20f4074_b1a5_4116_ac25_2d8d4235b7b7.slice/crio-c58a20b1277fdf78ca2e65a4053354b687b76f017e10245fbaf731cdea70fd5f WatchSource:0}: Error finding container c58a20b1277fdf78ca2e65a4053354b687b76f017e10245fbaf731cdea70fd5f: Status 404 returned error can't find the container with id c58a20b1277fdf78ca2e65a4053354b687b76f017e10245fbaf731cdea70fd5f Oct 30 08:53:33.044969 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:33.044941 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70cee2c6_bc37_4ab5_9f5f_5ded85c0e334.slice/crio-fa58aa796ef33d4bb7de2454907e70dc5379b2f40dbfd2930493687ad2f9fd65 WatchSource:0}: Error finding container fa58aa796ef33d4bb7de2454907e70dc5379b2f40dbfd2930493687ad2f9fd65: Status 404 returned error can't find the container with id fa58aa796ef33d4bb7de2454907e70dc5379b2f40dbfd2930493687ad2f9fd65 Oct 30 08:53:33.047647 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:33.047618 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f93427_45fe_46a3_8b9f_23d865534d41.slice/crio-295f318365e4eb99495f4a14fb0db2dd7ff26288014604e8c758f50e8d46cce5 WatchSource:0}: Error finding container 295f318365e4eb99495f4a14fb0db2dd7ff26288014604e8c758f50e8d46cce5: Status 404 returned error can't find the container with id 295f318365e4eb99495f4a14fb0db2dd7ff26288014604e8c758f50e8d46cce5 Oct 30 08:53:33.052537 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.052361 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:53:33.055113 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.055094 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:53:33.057817 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.057794 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:53:33.061803 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.061783 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:53:33.069429 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:33.069405 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b0e2a3_516f_4bf7_ba8f_a4dec22e4eb6.slice/crio-031cb85b6c9e79a498f26ef5470eacf148908a016f0fe2ca4c0806618779b30d WatchSource:0}: Error finding container 031cb85b6c9e79a498f26ef5470eacf148908a016f0fe2ca4c0806618779b30d: Status 404 returned error can't find the container with id 031cb85b6c9e79a498f26ef5470eacf148908a016f0fe2ca4c0806618779b30d Oct 30 08:53:33.073256 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:33.073223 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eebeab0_d5a7_443d_9ed8_365761816c98.slice/crio-231a4dc4aa0ceb519b5a4bb8c172528691b400defe5efdf373b307c25e222cdb WatchSource:0}: Error finding container 231a4dc4aa0ceb519b5a4bb8c172528691b400defe5efdf373b307c25e222cdb: Status 404 returned error can't find the container with id 231a4dc4aa0ceb519b5a4bb8c172528691b400defe5efdf373b307c25e222cdb Oct 30 08:53:33.075972 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:53:33.075944 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dee6e4b_1bc3_4718_a1a6_d268dd6b405c.slice/crio-16d5953fa5306400909cc7d6c8297df42e8021aad986ac6dc55711cdef0c32e8 WatchSource:0}: Error finding container 16d5953fa5306400909cc7d6c8297df42e8021aad986ac6dc55711cdef0c32e8: Status 404 returned error can't find the container with id 16d5953fa5306400909cc7d6c8297df42e8021aad986ac6dc55711cdef0c32e8 Oct 30 08:53:33.483132 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.482900 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrjj\" (UniqueName: \"kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj\") pod \"network-check-target-tf9r8\" (UID: \"25cb63c1-5df3-4be8-b407-b9bf2641cf95\") " pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:33.483235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.483168 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:33.483235 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:33.483193 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:33.483235 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:33.483213 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:33.483235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.483212 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:33.483235 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:33.483226 2578 projected.go:196] Error preparing data for projected volume kube-api-access-szrjj for pod openshift-network-diagnostics/network-check-target-tf9r8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:33.483399 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:33.483280 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj podName:25cb63c1-5df3-4be8-b407-b9bf2641cf95 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:34.483261023 +0000 UTC m=+4.236412612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-szrjj" (UniqueName: "kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj") pod "network-check-target-tf9r8" (UID: "25cb63c1-5df3-4be8-b407-b9bf2641cf95") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:33.483399 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:33.483343 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:33.483399 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:33.483343 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:33.483399 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:33.483380 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret podName:c7400965-54de-4420-a62a-37baf38d949d nodeName:}" failed. No retries permitted until 2026-10-30 08:53:34.48336793 +0000 UTC m=+4.236519518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret") pod "global-pull-secret-syncer-m5sz6" (UID: "c7400965-54de-4420-a62a-37baf38d949d") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:33.483399 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:33.483399 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs podName:f82405fd-bde5-4428-8b23-2be99042feef nodeName:}" failed. No retries permitted until 2026-10-30 08:53:34.483387656 +0000 UTC m=+4.236539229 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs") pod "network-metrics-daemon-ktt7j" (UID: "f82405fd-bde5-4428-8b23-2be99042feef") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:33.713894 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.713775 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 08:48:33 +0000 UTC" deadline="2028-06-18 12:43:48.059557995 +0000 UTC" Oct 30 08:53:33.713894 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.713848 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14331h50m14.345714047s" Oct 30 08:53:33.828287 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.828245 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nt944" event={"ID":"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c","Type":"ContainerStarted","Data":"16d5953fa5306400909cc7d6c8297df42e8021aad986ac6dc55711cdef0c32e8"} Oct 30 08:53:33.835496 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.835422 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q8z8s" event={"ID":"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6","Type":"ContainerStarted","Data":"031cb85b6c9e79a498f26ef5470eacf148908a016f0fe2ca4c0806618779b30d"} Oct 30 08:53:33.842196 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.842110 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kft2v" event={"ID":"87f93427-45fe-46a3-8b9f-23d865534d41","Type":"ContainerStarted","Data":"295f318365e4eb99495f4a14fb0db2dd7ff26288014604e8c758f50e8d46cce5"} Oct 30 08:53:33.848517 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.848422 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" event={"ID":"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334","Type":"ContainerStarted","Data":"fa58aa796ef33d4bb7de2454907e70dc5379b2f40dbfd2930493687ad2f9fd65"} Oct 30 08:53:33.858787 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.858656 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" event={"ID":"2b94c3af-6055-4f33-bd4b-57d4411d0b28","Type":"ContainerStarted","Data":"2a3783a7cc9d0b76dc3550f1418f8bc2e1d98f5ff5acc882644ad9dcdce5e69c"} Oct 30 08:53:33.868711 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.868503 2578 generic.go:356] "Generic (PLEG): container finished" podID="7f68f752438bee3aab3d894447acc3ea" containerID="b09aca77ac5af5ef49da1035110eea8ae02c22e96c7e08f7cf8ef11d21950ddc" exitCode=0 Oct 30 08:53:33.868711 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.868586 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" event={"ID":"7f68f752438bee3aab3d894447acc3ea","Type":"ContainerDied","Data":"b09aca77ac5af5ef49da1035110eea8ae02c22e96c7e08f7cf8ef11d21950ddc"} Oct 30 08:53:33.875001 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.874977 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" event={"ID":"9307784112acd4d13d34c64b57f18e71","Type":"ContainerStarted","Data":"b15d12f37e30e8e178ac969f728a57b6fd33b3661956e5a73f625546dd43ed35"} Oct 30 08:53:33.903761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.903730 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwg67" event={"ID":"2eebeab0-d5a7-443d-9ed8-365761816c98","Type":"ContainerStarted","Data":"231a4dc4aa0ceb519b5a4bb8c172528691b400defe5efdf373b307c25e222cdb"} Oct 30 08:53:33.913677 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.913621 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qgqjn" event={"ID":"d20f4074-b1a5-4116-ac25-2d8d4235b7b7","Type":"ContainerStarted","Data":"c58a20b1277fdf78ca2e65a4053354b687b76f017e10245fbaf731cdea70fd5f"} Oct 30 08:53:33.937811 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.937783 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hsqk7" event={"ID":"8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b","Type":"ContainerStarted","Data":"2dd179b53efad516bbc24bd56ba672e4606abc1638c1eeed9363a97b5cde81a8"} Oct 30 08:53:33.957569 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:33.957541 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mnq7z" event={"ID":"c032d4ff-047f-43a9-a27e-41affb60615d","Type":"ContainerStarted","Data":"d1884abd5bd8c6e79a33ddd9ed0fc18f5be091a41614d0942abdebf7bc5b5553"} Oct 30 08:53:34.494431 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:34.493152 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrjj\" (UniqueName: \"kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj\") pod \"network-check-target-tf9r8\" (UID: \"25cb63c1-5df3-4be8-b407-b9bf2641cf95\") " pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:34.494431 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:34.493229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:34.494431 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:34.493283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:34.494431 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:34.493463 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:34.494431 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:34.493534 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs podName:f82405fd-bde5-4428-8b23-2be99042feef nodeName:}" failed. No retries permitted until 2026-10-30 08:53:36.493513464 +0000 UTC m=+6.246665039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs") pod "network-metrics-daemon-ktt7j" (UID: "f82405fd-bde5-4428-8b23-2be99042feef") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:34.494431 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:34.494081 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:34.494431 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:34.494106 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:34.494431 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:34.494120 2578 projected.go:196] Error preparing data for projected volume kube-api-access-szrjj for pod openshift-network-diagnostics/network-check-target-tf9r8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:34.494431 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:34.494169 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj podName:25cb63c1-5df3-4be8-b407-b9bf2641cf95 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:36.494153617 +0000 UTC m=+6.247305199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-szrjj" (UniqueName: "kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj") pod "network-check-target-tf9r8" (UID: "25cb63c1-5df3-4be8-b407-b9bf2641cf95") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:34.494431 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:34.494241 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:34.494431 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:34.494281 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret podName:c7400965-54de-4420-a62a-37baf38d949d nodeName:}" failed. No retries permitted until 2026-10-30 08:53:36.4942687 +0000 UTC m=+6.247420278 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret") pod "global-pull-secret-syncer-m5sz6" (UID: "c7400965-54de-4420-a62a-37baf38d949d") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:34.714845 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:34.714798 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 08:48:33 +0000 UTC" deadline="2028-08-07 16:33:29.567676328 +0000 UTC" Oct 30 08:53:34.714845 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:34.714843 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15535h39m54.852838208s" Oct 30 08:53:34.805847 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:34.805784 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:34.806014 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:34.805971 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:53:34.806352 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:34.806326 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:34.806459 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:34.806413 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:53:34.806641 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:34.806584 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:34.806871 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:34.806749 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:53:34.975803 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:34.975763 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" event={"ID":"7f68f752438bee3aab3d894447acc3ea","Type":"ContainerStarted","Data":"56bd1ddca0149baca35b31eca5c5c77403b5084f722b9a38bd08930da5f25cbf"} Oct 30 08:53:35.016958 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:35.016208 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" podStartSLOduration=3.01618493 podStartE2EDuration="3.01618493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:33.948028313 +0000 UTC m=+3.701179909" watchObservedRunningTime="2026-10-30 08:53:35.01618493 +0000 UTC m=+4.769336526" Oct 30 08:53:35.987969 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:35.987190 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwg67" event={"ID":"2eebeab0-d5a7-443d-9ed8-365761816c98","Type":"ContainerStarted","Data":"944e4b896f61b55f39c727d0381bbeff779cba980d5f9a3cf572f1bdecb88822"} Oct 30 08:53:36.058530 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:36.058136 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" podStartSLOduration=4.058115506 podStartE2EDuration="4.058115506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:35.016924898 +0000 UTC m=+4.770076495" watchObservedRunningTime="2026-10-30 08:53:36.058115506 +0000 UTC m=+5.811267101" Oct 30 08:53:36.511030 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:36.510984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrjj\" (UniqueName: \"kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj\") pod \"network-check-target-tf9r8\" (UID: \"25cb63c1-5df3-4be8-b407-b9bf2641cf95\") " pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:36.511265 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:36.511051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:36.511265 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:36.511094 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:36.511265 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:36.511255 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:36.511467 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:36.511325 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs podName:f82405fd-bde5-4428-8b23-2be99042feef nodeName:}" failed. No retries permitted until 2026-10-30 08:53:40.511304729 +0000 UTC m=+10.264456320 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs") pod "network-metrics-daemon-ktt7j" (UID: "f82405fd-bde5-4428-8b23-2be99042feef") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:36.511840 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:36.511817 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:36.511840 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:36.511840 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:36.512032 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:36.511857 2578 projected.go:196] Error preparing data for projected volume kube-api-access-szrjj for pod openshift-network-diagnostics/network-check-target-tf9r8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:36.512032 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:36.511915 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj podName:25cb63c1-5df3-4be8-b407-b9bf2641cf95 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:40.511897678 +0000 UTC m=+10.265049254 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-szrjj" (UniqueName: "kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj") pod "network-check-target-tf9r8" (UID: "25cb63c1-5df3-4be8-b407-b9bf2641cf95") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:36.512032 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:36.511988 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:36.512032 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:36.512024 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret podName:c7400965-54de-4420-a62a-37baf38d949d nodeName:}" failed. No retries permitted until 2026-10-30 08:53:40.512012165 +0000 UTC m=+10.265163757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret") pod "global-pull-secret-syncer-m5sz6" (UID: "c7400965-54de-4420-a62a-37baf38d949d") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:36.805240 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:36.805150 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:36.805417 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:36.805295 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:53:36.805737 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:36.805715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:36.805853 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:36.805809 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:53:36.805923 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:36.805880 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:36.806014 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:36.805976 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:53:37.013022 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:37.012983 2578 generic.go:356] "Generic (PLEG): container finished" podID="2eebeab0-d5a7-443d-9ed8-365761816c98" containerID="944e4b896f61b55f39c727d0381bbeff779cba980d5f9a3cf572f1bdecb88822" exitCode=0 Oct 30 08:53:37.013514 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:37.013043 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwg67" event={"ID":"2eebeab0-d5a7-443d-9ed8-365761816c98","Type":"ContainerDied","Data":"944e4b896f61b55f39c727d0381bbeff779cba980d5f9a3cf572f1bdecb88822"} Oct 30 08:53:38.021355 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:38.020589 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hsqk7" event={"ID":"8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b","Type":"ContainerStarted","Data":"c272ba35a0478ba41ec919895a67378aa38e2c94dab40bba436d28f2340e3167"} Oct 30 08:53:38.150321 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:38.150128 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:53:38.151012 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:38.150784 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:53:38.192443 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:38.191331 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hsqk7" podStartSLOduration=3.101215192 podStartE2EDuration="7.1913106s" totalImagesPullingTime="4.090095408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:38.061784596 +0000 UTC m=+7.814936192" watchObservedRunningTime="2026-10-30 08:53:38.1913106 +0000 UTC m=+7.944462196" Oct 30 08:53:38.804807 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:38.804777 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:38.804933 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:38.804914 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:53:38.805311 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:38.805285 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:38.805420 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:38.805391 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:53:38.805484 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:38.805443 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:38.805535 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:38.805507 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:53:39.027872 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:39.027797 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mnq7z" event={"ID":"c032d4ff-047f-43a9-a27e-41affb60615d","Type":"ContainerStarted","Data":"0f11666b183def0622d28dd7e05a06ed0ff927b23c73e925a4da4fd86d3dfc51"} Oct 30 08:53:39.031404 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:39.031377 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" event={"ID":"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334","Type":"ContainerStarted","Data":"bf27654d6c25d93fa243aa53b62d2ba2ef09ce989c3eef8cc0c3cf490a967558"} Oct 30 08:53:39.031512 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:39.031438 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:53:39.032174 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:39.032157 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:53:39.112180 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:39.112120 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mnq7z" podStartSLOduration=2.941518131 podStartE2EDuration="8.112104563s" totalImagesPullingTime="5.170586432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:39.071670297 +0000 UTC m=+8.824821894" watchObservedRunningTime="2026-10-30 08:53:39.112104563 +0000 UTC m=+8.865256157" Oct 30 08:53:40.491804 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:40.491583 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 08:53:40.544171 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:40.544125 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:40.544341 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:40.544216 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrjj\" (UniqueName: \"kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj\") pod \"network-check-target-tf9r8\" (UID: \"25cb63c1-5df3-4be8-b407-b9bf2641cf95\") " pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:40.544341 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:40.544250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:40.544455 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:40.544378 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:40.544455 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:40.544451 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret podName:c7400965-54de-4420-a62a-37baf38d949d nodeName:}" failed. No retries permitted until 2026-10-30 08:53:48.544432701 +0000 UTC m=+18.297584292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret") pod "global-pull-secret-syncer-m5sz6" (UID: "c7400965-54de-4420-a62a-37baf38d949d") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:40.544922 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:40.544741 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:40.544922 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:40.544739 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:40.544922 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:40.544772 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:40.544922 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:40.544794 2578 projected.go:196] Error preparing data for projected volume kube-api-access-szrjj for pod openshift-network-diagnostics/network-check-target-tf9r8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:40.544922 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:40.544836 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs podName:f82405fd-bde5-4428-8b23-2be99042feef nodeName:}" failed. No retries permitted until 2026-10-30 08:53:48.544815491 +0000 UTC m=+18.297967079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs") pod "network-metrics-daemon-ktt7j" (UID: "f82405fd-bde5-4428-8b23-2be99042feef") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:40.544922 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:40.544864 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj podName:25cb63c1-5df3-4be8-b407-b9bf2641cf95 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:48.544844525 +0000 UTC m=+18.297996103 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-szrjj" (UniqueName: "kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj") pod "network-check-target-tf9r8" (UID: "25cb63c1-5df3-4be8-b407-b9bf2641cf95") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:40.726650 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:40.726506 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T08:53:40.4916501Z","UUID":"9bd553dd-9c61-4718-85d5-834064e0c399","Handler":null,"Name":"","Endpoint":""} Oct 30 08:53:40.728516 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:40.728493 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 08:53:40.728654 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:40.728525 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 08:53:40.803778 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:40.803743 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:40.803778 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:40.803771 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:40.803985 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:40.803865 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:53:40.805219 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:40.805196 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:40.805343 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:40.805310 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:53:40.805439 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:40.805413 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:53:41.049534 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:41.049015 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" event={"ID":"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334","Type":"ContainerStarted","Data":"5f794b059a41f83572e35bd8be6d9aa02c4371dc860c3ba290c78240599527ec"} Oct 30 08:53:42.054298 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:42.054072 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" event={"ID":"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334","Type":"ContainerStarted","Data":"f30a9d68bf8ecf91438838030be6f75e565ad97fc9a01870e488c452d615c184"} Oct 30 08:53:42.056461 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:42.056428 2578 generic.go:356] "Generic (PLEG): container finished" podID="2eebeab0-d5a7-443d-9ed8-365761816c98" containerID="dcb47eae769f35e34138619f2af8ef61ab1f7ecec947d2b3b8a71aadb6d8fadf" exitCode=0 Oct 30 08:53:42.056618 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:42.056498 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwg67" event={"ID":"2eebeab0-d5a7-443d-9ed8-365761816c98","Type":"ContainerDied","Data":"dcb47eae769f35e34138619f2af8ef61ab1f7ecec947d2b3b8a71aadb6d8fadf"} Oct 30 08:53:42.059341 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:42.059314 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qgqjn" event={"ID":"d20f4074-b1a5-4116-ac25-2d8d4235b7b7","Type":"ContainerStarted","Data":"ef6a29f91c533adaee47f74bd2974fed7f22d30a5a1caceafe9a29c608dd3b41"} Oct 30 08:53:42.061191 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:42.061151 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q8z8s" event={"ID":"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6","Type":"ContainerStarted","Data":"49b7ab0f714e845bbb1068cc8b871d3bc62a4a834e84edf5e9119d3c7206c50c"} Oct 30 08:53:42.097007 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:42.096963 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" podStartSLOduration=2.785932792 podStartE2EDuration="11.09695152s" totalImagesPullingTime="8.311018728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:42.095943349 +0000 UTC m=+11.849094961" watchObservedRunningTime="2026-10-30 08:53:42.09695152 +0000 UTC m=+11.850103115" Oct 30 08:53:42.180826 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:42.180778 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qgqjn" podStartSLOduration=3.243831826 podStartE2EDuration="11.180763491s" totalImagesPullingTime="7.936931665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:42.180343282 +0000 UTC m=+11.933494889" watchObservedRunningTime="2026-10-30 08:53:42.180763491 +0000 UTC m=+11.933915085" Oct 30 08:53:42.220718 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:42.219433 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-q8z8s" podStartSLOduration=2.736150673 podStartE2EDuration="11.21942022s" totalImagesPullingTime="8.483269547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:42.219075034 +0000 UTC m=+11.972226630" watchObservedRunningTime="2026-10-30 08:53:42.21942022 +0000 UTC m=+11.972571819" Oct 30 08:53:42.802634 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:42.802586 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:42.802840 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:42.802591 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:42.802840 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:42.802741 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:53:42.802840 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:42.802761 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:42.803019 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:42.802866 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:53:42.803019 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:42.802946 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:53:43.067528 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:43.067492 2578 generic.go:356] "Generic (PLEG): container finished" podID="2eebeab0-d5a7-443d-9ed8-365761816c98" containerID="d1ac84a97e6eb5b19831f97a86e4c4ea77354a78d65c0316689661ca8df92290" exitCode=0 Oct 30 08:53:43.068190 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:43.067579 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwg67" event={"ID":"2eebeab0-d5a7-443d-9ed8-365761816c98","Type":"ContainerDied","Data":"d1ac84a97e6eb5b19831f97a86e4c4ea77354a78d65c0316689661ca8df92290"} Oct 30 08:53:43.070277 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:43.070254 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kft2v" event={"ID":"87f93427-45fe-46a3-8b9f-23d865534d41","Type":"ContainerStarted","Data":"d54282067196a83e5bc587ed356249694473f9a87331289b3d9d0f1425a6c665"} Oct 30 08:53:43.153065 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:43.153010 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kft2v" podStartSLOduration=3.779949957 podStartE2EDuration="12.152996132s" totalImagesPullingTime="8.373046175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:43.15221317 +0000 UTC m=+12.905364768" watchObservedRunningTime="2026-10-30 08:53:43.152996132 +0000 UTC m=+12.906147775" Oct 30 08:53:44.074649 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:44.074562 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nt944" event={"ID":"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c","Type":"ContainerStarted","Data":"25528d74035b6b602e06b4a54fee08e4df4bde99c7523f6ab057a754a4e12f4c"} Oct 30 08:53:44.112444 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:44.112391 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nt944" podStartSLOduration=2.3811642170000002 podStartE2EDuration="13.112373689s" totalImagesPullingTime="10.731209472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:44.112267859 +0000 UTC m=+13.865419450" watchObservedRunningTime="2026-10-30 08:53:44.112373689 +0000 UTC m=+13.865525281" Oct 30 08:53:44.803128 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:44.803092 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:44.803280 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:44.803132 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:44.803280 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:44.803201 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:53:44.803352 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:44.803281 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:44.803424 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:44.803397 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:53:44.803521 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:44.803499 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:53:45.078931 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:45.078851 2578 generic.go:356] "Generic (PLEG): container finished" podID="2eebeab0-d5a7-443d-9ed8-365761816c98" containerID="b7c0a73e793951aac6b9aa847adc18c5d866e14950a5a85eff4576666119faff" exitCode=0 Oct 30 08:53:45.079498 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:45.079464 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwg67" event={"ID":"2eebeab0-d5a7-443d-9ed8-365761816c98","Type":"ContainerDied","Data":"b7c0a73e793951aac6b9aa847adc18c5d866e14950a5a85eff4576666119faff"} Oct 30 08:53:46.802946 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:46.802743 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:46.803437 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:46.802741 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:46.803437 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:46.803028 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:53:46.803437 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:46.803115 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:53:46.803437 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:46.802818 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:46.803437 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:46.803225 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:53:47.087079 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:47.087047 2578 generic.go:356] "Generic (PLEG): container finished" podID="2eebeab0-d5a7-443d-9ed8-365761816c98" containerID="c43d32e9fc013d3c2ffa6ba5fa6a3a5826f08692ebeac5b6169415b255edf233" exitCode=0 Oct 30 08:53:47.087186 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:47.087094 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwg67" event={"ID":"2eebeab0-d5a7-443d-9ed8-365761816c98","Type":"ContainerDied","Data":"c43d32e9fc013d3c2ffa6ba5fa6a3a5826f08692ebeac5b6169415b255edf233"} Oct 30 08:53:48.091935 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:48.091900 2578 generic.go:356] "Generic (PLEG): container finished" podID="2eebeab0-d5a7-443d-9ed8-365761816c98" containerID="46a21a8ba482075246800ea45b1781f7ec06b7aecc30afc9e65b7368af39455b" exitCode=0 Oct 30 08:53:48.092301 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:48.091955 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwg67" event={"ID":"2eebeab0-d5a7-443d-9ed8-365761816c98","Type":"ContainerDied","Data":"46a21a8ba482075246800ea45b1781f7ec06b7aecc30afc9e65b7368af39455b"} Oct 30 08:53:48.604439 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:48.604312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrjj\" (UniqueName: \"kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj\") pod \"network-check-target-tf9r8\" (UID: \"25cb63c1-5df3-4be8-b407-b9bf2641cf95\") " pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:48.604535 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:48.604449 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:48.604535 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:48.604458 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:48.604535 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:48.604465 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:48.604535 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:48.604499 2578 projected.go:196] Error preparing data for projected volume kube-api-access-szrjj for pod openshift-network-diagnostics/network-check-target-tf9r8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:48.604535 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:48.604519 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:48.604535 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:48.604503 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:48.604779 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:48.604544 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj podName:25cb63c1-5df3-4be8-b407-b9bf2641cf95 nodeName:}" failed. No retries permitted until 2026-10-30 08:54:04.604531145 +0000 UTC m=+34.357682729 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-szrjj" (UniqueName: "kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj") pod "network-check-target-tf9r8" (UID: "25cb63c1-5df3-4be8-b407-b9bf2641cf95") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:48.604779 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:48.604559 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:48.604779 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:48.604605 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret podName:c7400965-54de-4420-a62a-37baf38d949d nodeName:}" failed. No retries permitted until 2026-10-30 08:54:04.604567862 +0000 UTC m=+34.357719434 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret") pod "global-pull-secret-syncer-m5sz6" (UID: "c7400965-54de-4420-a62a-37baf38d949d") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:48.604779 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:48.604628 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs podName:f82405fd-bde5-4428-8b23-2be99042feef nodeName:}" failed. No retries permitted until 2026-10-30 08:54:04.604619914 +0000 UTC m=+34.357771482 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs") pod "network-metrics-daemon-ktt7j" (UID: "f82405fd-bde5-4428-8b23-2be99042feef") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:48.802369 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:48.802341 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:48.802488 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:48.802345 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:48.802488 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:48.802456 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:48.802488 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:48.802450 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:53:48.802686 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:48.802545 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:53:48.802686 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:48.802658 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:53:49.097166 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:49.097108 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwg67" event={"ID":"2eebeab0-d5a7-443d-9ed8-365761816c98","Type":"ContainerStarted","Data":"59aa1bc55247e62f73ed89ed81e2ed1f70106092ebbee85b9ba5bda6f2ec5f03"} Oct 30 08:53:49.099863 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:49.099841 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" event={"ID":"2b94c3af-6055-4f33-bd4b-57d4411d0b28","Type":"ContainerStarted","Data":"6c87ffe4d76fe659e030327aacb3e8fc3b880abb5d5c49a0798161a195d797f0"} Oct 30 08:53:49.099952 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:49.099900 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" event={"ID":"2b94c3af-6055-4f33-bd4b-57d4411d0b28","Type":"ContainerStarted","Data":"0e535db964ec94d66c85f13bbf1f6a452f08ae748529fb71647de3986378459b"} Oct 30 08:53:49.099952 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:49.099932 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" event={"ID":"2b94c3af-6055-4f33-bd4b-57d4411d0b28","Type":"ContainerStarted","Data":"7137defda4c93eb40e52b2564eb688687c1c4f1df0d9af277c3741f3f9fa55c9"} Oct 30 08:53:49.100024 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:49.099963 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" event={"ID":"2b94c3af-6055-4f33-bd4b-57d4411d0b28","Type":"ContainerStarted","Data":"5e1cfea03d0a7b480651666a24c9c735a00db238b9ca0da24e3e7a4cbbe60d34"} Oct 30 08:53:49.100024 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:49.099991 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" event={"ID":"2b94c3af-6055-4f33-bd4b-57d4411d0b28","Type":"ContainerStarted","Data":"8a18f4cc3f01678b6b07e11d3877b57f374cf4d0dd678f261a8e06fd336d4364"} Oct 30 08:53:49.100076 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:49.100025 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" event={"ID":"2b94c3af-6055-4f33-bd4b-57d4411d0b28","Type":"ContainerStarted","Data":"30b716e272c97812da1b38d561b0c93443146980db4c22cb85a3926e09d9363b"} Oct 30 08:53:49.138192 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:49.138154 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hwg67" podStartSLOduration=7.43965953 podStartE2EDuration="18.138143443s" totalImagesPullingTime="10.698483913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:49.137675518 +0000 UTC m=+18.890827111" watchObservedRunningTime="2026-10-30 08:53:49.138143443 +0000 UTC m=+18.891295033" Oct 30 08:53:50.803551 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:50.803525 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:50.804019 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:50.803711 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:50.804019 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:50.803734 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:53:50.804019 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:50.803839 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:50.804019 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:50.803914 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:53:50.804019 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:50.803953 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:53:51.106504 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:51.106434 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" event={"ID":"2b94c3af-6055-4f33-bd4b-57d4411d0b28","Type":"ContainerStarted","Data":"755504e9f1438bcdaa60657b629fc3206e620ba7e1774f344e12f60a967dfe0c"} Oct 30 08:53:52.802689 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:52.802659 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:52.803132 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:52.802668 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:52.803132 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:52.802752 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:53:52.803132 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:52.802764 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:52.803132 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:52.802869 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:53:52.803132 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:52.802965 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:53:53.113635 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:53.113535 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" event={"ID":"2b94c3af-6055-4f33-bd4b-57d4411d0b28","Type":"ContainerStarted","Data":"f87583d1b771b29ecc331a87b13991d71366f4a3a32ed91b477c96ac155d9938"} Oct 30 08:53:53.113784 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:53.113767 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:53.129424 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:53.129401 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:53.196835 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:53.196791 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" podStartSLOduration=6.835721542 podStartE2EDuration="22.196779487s" totalImagesPullingTime="15.361057945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:53.183983117 +0000 UTC m=+22.937134708" watchObservedRunningTime="2026-10-30 08:53:53.196779487 +0000 UTC m=+22.949931077" Oct 30 08:53:54.115863 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:54.115645 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 08:53:54.116441 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:54.115818 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:54.131327 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:54.131303 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:53:54.802279 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:54.802248 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:54.802450 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:54.802296 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:54.802450 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:54.802313 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:54.802450 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:54.802405 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:53:54.802639 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:54.802472 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:53:54.802639 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:54.802578 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:53:55.117699 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:55.117675 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 08:53:56.119543 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:56.119516 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 08:53:56.802658 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:56.802627 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:56.802658 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:56.802644 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:56.802888 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:56.802627 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:56.802888 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:56.802738 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:53:56.802888 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:56.802803 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:53:56.803023 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:56.802896 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:53:57.484795 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:57.484769 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q8z8s_b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6/dns-node-resolver/0.log" Oct 30 08:53:58.230962 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:58.230936 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mnq7z_c032d4ff-047f-43a9-a27e-41affb60615d/node-ca/0.log" Oct 30 08:53:58.803198 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:58.803162 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:53:58.803198 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:58.803181 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:53:58.803670 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:53:58.803284 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:53:58.803670 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:58.803290 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:53:58.803670 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:58.803369 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:53:58.803670 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:53:58.803465 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:54:00.233647 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:00.233611 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:00.234088 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:00.233853 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 08:54:00.243856 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:00.243821 2578 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" podUID="2b94c3af-6055-4f33-bd4b-57d4411d0b28" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 08:54:00.252179 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:00.252152 2578 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" podUID="2b94c3af-6055-4f33-bd4b-57d4411d0b28" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 08:54:00.803137 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:00.803112 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:00.803314 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:00.803224 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:54:00.803370 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:00.803312 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:00.803418 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:00.803398 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:00.803453 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:00.803417 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:54:00.803505 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:00.803462 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:54:02.802978 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:02.802943 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:02.802978 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:02.802975 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:02.803482 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:02.802943 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:02.803482 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:02.803056 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:54:02.803482 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:02.803111 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:54:02.803482 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:02.803189 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:54:04.623241 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:04.623205 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrjj\" (UniqueName: \"kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj\") pod \"network-check-target-tf9r8\" (UID: \"25cb63c1-5df3-4be8-b407-b9bf2641cf95\") " pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:04.623681 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:04.623260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:04.623681 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:04.623301 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:04.623681 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:04.623321 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:54:04.623681 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:04.623359 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:54:04.623681 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:04.623372 2578 projected.go:196] Error preparing data for projected volume kube-api-access-szrjj for pod openshift-network-diagnostics/network-check-target-tf9r8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:54:04.623681 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:04.623378 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:54:04.623681 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:04.623392 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:54:04.623681 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:04.623428 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj podName:25cb63c1-5df3-4be8-b407-b9bf2641cf95 nodeName:}" failed. No retries permitted until 2026-10-30 08:54:36.623413289 +0000 UTC m=+66.376564863 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-szrjj" (UniqueName: "kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj") pod "network-check-target-tf9r8" (UID: "25cb63c1-5df3-4be8-b407-b9bf2641cf95") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:54:04.623681 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:04.623449 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret podName:c7400965-54de-4420-a62a-37baf38d949d nodeName:}" failed. No retries permitted until 2026-10-30 08:54:36.623440897 +0000 UTC m=+66.376592466 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret") pod "global-pull-secret-syncer-m5sz6" (UID: "c7400965-54de-4420-a62a-37baf38d949d") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:54:04.623681 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:04.623463 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs podName:f82405fd-bde5-4428-8b23-2be99042feef nodeName:}" failed. No retries permitted until 2026-10-30 08:54:36.623454953 +0000 UTC m=+66.376606522 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs") pod "network-metrics-daemon-ktt7j" (UID: "f82405fd-bde5-4428-8b23-2be99042feef") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:54:04.802721 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:04.802690 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:04.802838 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:04.802692 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:04.802838 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:04.802797 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:04.802838 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:04.802797 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:54:04.802950 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:04.802891 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:54:04.802983 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:04.802972 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:54:06.802808 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:06.802771 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:06.803210 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:06.802835 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:06.803210 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:06.802843 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:06.803210 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:06.802967 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:54:06.803210 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:06.803084 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:54:06.803210 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:06.803159 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:54:08.802943 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:08.802912 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:08.803367 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:08.802943 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:08.803367 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:08.803032 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:54:08.803367 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:08.803080 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:54:08.803367 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:08.803123 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:08.803367 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:08.803197 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:54:10.803057 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:10.803023 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:10.803472 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:10.803153 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:10.803472 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:10.803177 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:54:10.803472 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:10.803260 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:54:10.803472 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:10.803288 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:10.803472 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:10.803415 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:54:12.803308 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:12.803265 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:12.803308 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:12.803285 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:12.803308 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:12.803295 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:12.803799 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:12.803400 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:54:12.803799 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:12.803545 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:54:12.803799 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:12.803662 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:54:12.961038 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:12.961012 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tf9r8"] Oct 30 08:54:12.964681 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:12.964649 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m5sz6"] Oct 30 08:54:12.965271 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:12.965248 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ktt7j"] Oct 30 08:54:13.154473 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:13.154443 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:13.154473 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:13.154472 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:13.154653 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:13.154555 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:54:13.154653 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:13.154578 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:13.154728 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:13.154681 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:54:13.154768 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:13.154734 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:54:14.803016 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:14.802989 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:14.803660 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:14.802991 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:14.803660 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:14.803083 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:54:14.803660 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:14.803170 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:54:14.803660 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:14.803215 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:14.803660 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:14.803290 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:54:16.802704 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:16.802541 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:16.803167 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:16.802551 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:16.803167 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:16.802771 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tf9r8" podUID="25cb63c1-5df3-4be8-b407-b9bf2641cf95" Oct 30 08:54:16.803167 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:16.802861 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ktt7j" podUID="f82405fd-bde5-4428-8b23-2be99042feef" Oct 30 08:54:16.803167 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:16.802575 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:16.803167 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:16.802948 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5sz6" podUID="c7400965-54de-4420-a62a-37baf38d949d" Oct 30 08:54:17.592779 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.592754 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeReady" Oct 30 08:54:17.592910 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.592886 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 08:54:17.648726 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.648668 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fxq45"] Oct 30 08:54:17.651792 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.651772 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kwlz8"] Oct 30 08:54:17.651904 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.651889 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:17.654552 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.654529 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kwlz8" Oct 30 08:54:17.665142 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.665125 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vbz6m"] Oct 30 08:54:17.667843 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.667827 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:17.670483 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.670464 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 08:54:17.670483 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.670465 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xqvv4\"" Oct 30 08:54:17.670607 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.670508 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 08:54:17.671128 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.671104 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kwlz8"] Oct 30 08:54:17.677036 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.676927 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 08:54:17.677036 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.676934 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 08:54:17.677036 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.677008 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 08:54:17.680743 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.680726 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cb5mw\"" Oct 30 08:54:17.681608 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.681574 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fxq45"] Oct 30 08:54:17.688869 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.688851 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vbz6m"] Oct 30 08:54:17.770452 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.770412 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 08:54:17.770452 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.770436 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 08:54:17.770452 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.770438 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 08:54:17.770734 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.770564 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wqhvd\"" Oct 30 08:54:17.770734 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.770668 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 08:54:17.821977 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.821954 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f64a1d53-38a0-4133-accb-411e2dc6fc64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:17.822319 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.821990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f64a1d53-38a0-4133-accb-411e2dc6fc64-crio-socket\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:17.822319 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.822009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f64a1d53-38a0-4133-accb-411e2dc6fc64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:17.822319 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.822070 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f76551a8-8b0b-459d-86f5-84cf822e83b8-tmp-dir\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:17.822319 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.822101 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f76551a8-8b0b-459d-86f5-84cf822e83b8-metrics-tls\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:17.822319 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.822116 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab3d086c-96af-4210-abf6-0cfbfe3afb20-cert\") pod \"ingress-canary-kwlz8\" (UID: \"ab3d086c-96af-4210-abf6-0cfbfe3afb20\") " pod="openshift-ingress-canary/ingress-canary-kwlz8" Oct 30 08:54:17.822319 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.822140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hj9q\" (UniqueName: \"kubernetes.io/projected/f76551a8-8b0b-459d-86f5-84cf822e83b8-kube-api-access-2hj9q\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:17.822319 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.822239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgcgt\" (UniqueName: \"kubernetes.io/projected/f64a1d53-38a0-4133-accb-411e2dc6fc64-kube-api-access-tgcgt\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:17.822319 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.822294 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f64a1d53-38a0-4133-accb-411e2dc6fc64-data-volume\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:17.822576 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.822349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjskv\" (UniqueName: \"kubernetes.io/projected/ab3d086c-96af-4210-abf6-0cfbfe3afb20-kube-api-access-sjskv\") pod \"ingress-canary-kwlz8\" (UID: \"ab3d086c-96af-4210-abf6-0cfbfe3afb20\") " pod="openshift-ingress-canary/ingress-canary-kwlz8" Oct 30 08:54:17.822576 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.822406 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f76551a8-8b0b-459d-86f5-84cf822e83b8-config-volume\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:17.923281 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.923236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hj9q\" (UniqueName: \"kubernetes.io/projected/f76551a8-8b0b-459d-86f5-84cf822e83b8-kube-api-access-2hj9q\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:17.923281 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.923262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgcgt\" (UniqueName: \"kubernetes.io/projected/f64a1d53-38a0-4133-accb-411e2dc6fc64-kube-api-access-tgcgt\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:17.923414 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.923284 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f64a1d53-38a0-4133-accb-411e2dc6fc64-data-volume\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:17.923414 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.923305 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjskv\" (UniqueName: \"kubernetes.io/projected/ab3d086c-96af-4210-abf6-0cfbfe3afb20-kube-api-access-sjskv\") pod \"ingress-canary-kwlz8\" (UID: \"ab3d086c-96af-4210-abf6-0cfbfe3afb20\") " pod="openshift-ingress-canary/ingress-canary-kwlz8" Oct 30 08:54:17.923512 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.923418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f76551a8-8b0b-459d-86f5-84cf822e83b8-config-volume\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:17.923512 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.923454 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f64a1d53-38a0-4133-accb-411e2dc6fc64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:17.923512 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.923490 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f64a1d53-38a0-4133-accb-411e2dc6fc64-crio-socket\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:17.923512 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.923506 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f64a1d53-38a0-4133-accb-411e2dc6fc64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:17.923777 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.923569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f76551a8-8b0b-459d-86f5-84cf822e83b8-tmp-dir\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:17.923777 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.923632 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f76551a8-8b0b-459d-86f5-84cf822e83b8-metrics-tls\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:17.923777 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.923652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab3d086c-96af-4210-abf6-0cfbfe3afb20-cert\") pod \"ingress-canary-kwlz8\" (UID: \"ab3d086c-96af-4210-abf6-0cfbfe3afb20\") " pod="openshift-ingress-canary/ingress-canary-kwlz8" Oct 30 08:54:17.923777 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.923668 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f64a1d53-38a0-4133-accb-411e2dc6fc64-data-volume\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:17.923777 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.923734 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f64a1d53-38a0-4133-accb-411e2dc6fc64-crio-socket\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:17.924006 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.923946 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f76551a8-8b0b-459d-86f5-84cf822e83b8-tmp-dir\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:17.924129 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.924105 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f64a1d53-38a0-4133-accb-411e2dc6fc64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:17.924182 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.924116 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f76551a8-8b0b-459d-86f5-84cf822e83b8-config-volume\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:17.927506 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.927475 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f76551a8-8b0b-459d-86f5-84cf822e83b8-metrics-tls\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:17.927506 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.927491 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f64a1d53-38a0-4133-accb-411e2dc6fc64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:17.927651 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.927510 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab3d086c-96af-4210-abf6-0cfbfe3afb20-cert\") pod \"ingress-canary-kwlz8\" (UID: \"ab3d086c-96af-4210-abf6-0cfbfe3afb20\") " pod="openshift-ingress-canary/ingress-canary-kwlz8" Oct 30 08:54:17.942486 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.942459 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgcgt\" (UniqueName: \"kubernetes.io/projected/f64a1d53-38a0-4133-accb-411e2dc6fc64-kube-api-access-tgcgt\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:17.943205 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.943184 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hj9q\" (UniqueName: \"kubernetes.io/projected/f76551a8-8b0b-459d-86f5-84cf822e83b8-kube-api-access-2hj9q\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:17.943481 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.943465 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjskv\" (UniqueName: \"kubernetes.io/projected/ab3d086c-96af-4210-abf6-0cfbfe3afb20-kube-api-access-sjskv\") pod \"ingress-canary-kwlz8\" (UID: \"ab3d086c-96af-4210-abf6-0cfbfe3afb20\") " pod="openshift-ingress-canary/ingress-canary-kwlz8" Oct 30 08:54:17.965894 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.965879 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:17.969014 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.968997 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kwlz8" Oct 30 08:54:17.984011 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:17.983993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:18.275880 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.275813 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kwlz8"] Oct 30 08:54:18.281023 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:54:18.280999 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3d086c_96af_4210_abf6_0cfbfe3afb20.slice/crio-afe2b09f39c62c80133b72645d1be31061be5cc1e171d66b4cb0067ce45d46b0 WatchSource:0}: Error finding container afe2b09f39c62c80133b72645d1be31061be5cc1e171d66b4cb0067ce45d46b0: Status 404 returned error can't find the container with id afe2b09f39c62c80133b72645d1be31061be5cc1e171d66b4cb0067ce45d46b0 Oct 30 08:54:18.288878 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.288856 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fxq45"] Oct 30 08:54:18.293174 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:54:18.293153 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76551a8_8b0b_459d_86f5_84cf822e83b8.slice/crio-16e37ee9a32fff5b4adb761e178a3c321d8cb83004ac2911e5a304475f8947c2 WatchSource:0}: Error finding container 16e37ee9a32fff5b4adb761e178a3c321d8cb83004ac2911e5a304475f8947c2: Status 404 returned error can't find the container with id 16e37ee9a32fff5b4adb761e178a3c321d8cb83004ac2911e5a304475f8947c2 Oct 30 08:54:18.302842 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.302824 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vbz6m"] Oct 30 08:54:18.306862 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:54:18.306841 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64a1d53_38a0_4133_accb_411e2dc6fc64.slice/crio-cf6dd05fa5152d6207f48f8a0c192b5d84b9b8f376bd4300bc351fa6ced996db WatchSource:0}: Error finding container cf6dd05fa5152d6207f48f8a0c192b5d84b9b8f376bd4300bc351fa6ced996db: Status 404 returned error can't find the container with id cf6dd05fa5152d6207f48f8a0c192b5d84b9b8f376bd4300bc351fa6ced996db Oct 30 08:54:18.486811 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.486785 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-z42cw"] Oct 30 08:54:18.491731 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.491705 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:18.512338 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.512295 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 08:54:18.512338 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.512332 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 08:54:18.512512 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.512358 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 08:54:18.512605 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.512576 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 08:54:18.512647 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.512609 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-6sdrk\"" Oct 30 08:54:18.512711 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.512696 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 08:54:18.513146 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.513124 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-z42cw"] Oct 30 08:54:18.628740 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.628717 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e23f66da-2587-4ce8-bb3a-a344e5dad910-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:18.628843 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.628756 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frlr5\" (UniqueName: \"kubernetes.io/projected/e23f66da-2587-4ce8-bb3a-a344e5dad910-kube-api-access-frlr5\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:18.628882 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.628853 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e23f66da-2587-4ce8-bb3a-a344e5dad910-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:18.628920 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.628886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e23f66da-2587-4ce8-bb3a-a344e5dad910-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:18.729988 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.729964 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e23f66da-2587-4ce8-bb3a-a344e5dad910-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:18.730132 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.730002 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e23f66da-2587-4ce8-bb3a-a344e5dad910-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:18.730132 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.730074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e23f66da-2587-4ce8-bb3a-a344e5dad910-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:18.730132 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.730103 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frlr5\" (UniqueName: \"kubernetes.io/projected/e23f66da-2587-4ce8-bb3a-a344e5dad910-kube-api-access-frlr5\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:18.731288 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.731262 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e23f66da-2587-4ce8-bb3a-a344e5dad910-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:18.732946 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.732920 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e23f66da-2587-4ce8-bb3a-a344e5dad910-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:18.733059 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.733031 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e23f66da-2587-4ce8-bb3a-a344e5dad910-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:18.749645 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.749622 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frlr5\" (UniqueName: \"kubernetes.io/projected/e23f66da-2587-4ce8-bb3a-a344e5dad910-kube-api-access-frlr5\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:18.802926 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.802898 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:18.803330 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.803305 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:18.803745 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.803722 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:18.804519 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.804505 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:18.819086 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.818870 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 08:54:18.819086 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.818875 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 08:54:18.819086 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.819047 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7pnl7\"" Oct 30 08:54:18.819362 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.819084 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kgscj\"" Oct 30 08:54:18.819362 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.819103 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 08:54:18.819362 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.819146 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r2tvf\"" Oct 30 08:54:18.819362 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:18.819176 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 08:54:19.115990 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:19.115960 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-z42cw"] Oct 30 08:54:19.124031 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:54:19.123998 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23f66da_2587_4ce8_bb3a_a344e5dad910.slice/crio-479a3215da80ed97eff8e2de3caf716c5bfae487c1b71dca238320902c5dae59 WatchSource:0}: Error finding container 479a3215da80ed97eff8e2de3caf716c5bfae487c1b71dca238320902c5dae59: Status 404 returned error can't find the container with id 479a3215da80ed97eff8e2de3caf716c5bfae487c1b71dca238320902c5dae59 Oct 30 08:54:19.169064 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:19.168998 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kwlz8" event={"ID":"ab3d086c-96af-4210-abf6-0cfbfe3afb20","Type":"ContainerStarted","Data":"afe2b09f39c62c80133b72645d1be31061be5cc1e171d66b4cb0067ce45d46b0"} Oct 30 08:54:19.170209 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:19.170161 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" event={"ID":"e23f66da-2587-4ce8-bb3a-a344e5dad910","Type":"ContainerStarted","Data":"479a3215da80ed97eff8e2de3caf716c5bfae487c1b71dca238320902c5dae59"} Oct 30 08:54:19.171481 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:19.171434 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fxq45" event={"ID":"f76551a8-8b0b-459d-86f5-84cf822e83b8","Type":"ContainerStarted","Data":"16e37ee9a32fff5b4adb761e178a3c321d8cb83004ac2911e5a304475f8947c2"} Oct 30 08:54:19.173854 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:19.173829 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vbz6m" event={"ID":"f64a1d53-38a0-4133-accb-411e2dc6fc64","Type":"ContainerStarted","Data":"ced9e506e20dfe450808717d75be719a6a159a8333799850f92a5288d54b98a7"} Oct 30 08:54:19.173921 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:19.173885 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vbz6m" event={"ID":"f64a1d53-38a0-4133-accb-411e2dc6fc64","Type":"ContainerStarted","Data":"cf6dd05fa5152d6207f48f8a0c192b5d84b9b8f376bd4300bc351fa6ced996db"} Oct 30 08:54:20.179532 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:20.179453 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vbz6m" event={"ID":"f64a1d53-38a0-4133-accb-411e2dc6fc64","Type":"ContainerStarted","Data":"c3addb8a4ba350d4092f9c4df396e5c2d1ce9c97b8db9f078e9925e92828ead5"} Oct 30 08:54:21.185361 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:21.185254 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" event={"ID":"e23f66da-2587-4ce8-bb3a-a344e5dad910","Type":"ContainerStarted","Data":"881e3ef25f0af4c713faaf6d2a5dac1f3918756e9ccdcb2bea3b9064de536d51"} Oct 30 08:54:21.185361 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:21.185318 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" event={"ID":"e23f66da-2587-4ce8-bb3a-a344e5dad910","Type":"ContainerStarted","Data":"8e8550e1d0cb4c0efbe08c445ab4a079e742a4042eaaf97b237900dedacdb674"} Oct 30 08:54:21.187794 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:21.187770 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fxq45" event={"ID":"f76551a8-8b0b-459d-86f5-84cf822e83b8","Type":"ContainerStarted","Data":"87d7d0c8845fb17cf7dd202b6774982586eea36d6388da2200ae043a5994dc14"} Oct 30 08:54:21.187940 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:21.187824 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:21.187940 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:21.187847 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fxq45" event={"ID":"f76551a8-8b0b-459d-86f5-84cf822e83b8","Type":"ContainerStarted","Data":"de1ecef0605c3f59150e7819887e44849d8f55a9dc821c8ab120fba78992523f"} Oct 30 08:54:21.190315 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:21.190288 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vbz6m" event={"ID":"f64a1d53-38a0-4133-accb-411e2dc6fc64","Type":"ContainerStarted","Data":"49c6d28f5635b152aa2fb82baed7592df5cb361542613a79ca3e171f98852bbc"} Oct 30 08:54:21.235517 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:21.235445 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-fxq45" podStartSLOduration=2.246187124 podStartE2EDuration="4.235433769s" totalImagesPullingTime="1.989246645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:21.234743426 +0000 UTC m=+50.987895019" watchObservedRunningTime="2026-10-30 08:54:21.235433769 +0000 UTC m=+50.988585359" Oct 30 08:54:21.273968 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:21.273933 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vbz6m" podStartSLOduration=1.7773175380000001 podStartE2EDuration="4.273921345s" totalImagesPullingTime="2.496603807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:54:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:21.273044829 +0000 UTC m=+51.026196421" watchObservedRunningTime="2026-10-30 08:54:21.273921345 +0000 UTC m=+51.027072936" Oct 30 08:54:21.310752 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:21.310702 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" podStartSLOduration=1.784858887 podStartE2EDuration="3.310685663s" totalImagesPullingTime="1.525826776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:21.309946815 +0000 UTC m=+51.063098406" watchObservedRunningTime="2026-10-30 08:54:21.310685663 +0000 UTC m=+51.063837255" Oct 30 08:54:22.197691 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:22.197655 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kwlz8" event={"ID":"ab3d086c-96af-4210-abf6-0cfbfe3afb20","Type":"ContainerStarted","Data":"04f018eed918a76dc84378761ae876ec36ef8465bd5a7ccd5f29eb86833a1a8a"} Oct 30 08:54:22.237465 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:22.237425 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kwlz8" podStartSLOduration=2.260503412 podStartE2EDuration="5.237413599s" totalImagesPullingTime="2.976910187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:22.236361648 +0000 UTC m=+51.989513238" watchObservedRunningTime="2026-10-30 08:54:22.237413599 +0000 UTC m=+51.990565189" Oct 30 08:54:23.060273 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.060194 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2"] Oct 30 08:54:23.063414 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.063398 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:23.092347 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.092325 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 08:54:23.092347 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.092327 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 08:54:23.092520 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.092406 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hr7xt\"" Oct 30 08:54:23.093444 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.093423 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2"] Oct 30 08:54:23.103489 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.103461 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg"] Oct 30 08:54:23.106607 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.106575 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.107368 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.107349 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8kgvx"] Oct 30 08:54:23.110270 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.110252 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.121868 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.121852 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 08:54:23.127191 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.127162 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg"] Oct 30 08:54:23.127599 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.127564 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tpklt\"" Oct 30 08:54:23.127707 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.127610 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 08:54:23.127707 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.127616 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 08:54:23.127707 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.127640 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 08:54:23.127707 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.127683 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 08:54:23.129069 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.129051 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jptkp\"" Oct 30 08:54:23.166022 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.165968 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/866d9622-0c68-4e07-94ef-94cabbd34f86-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:23.166022 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.166012 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/866d9622-0c68-4e07-94ef-94cabbd34f86-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:23.166191 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.166045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/866d9622-0c68-4e07-94ef-94cabbd34f86-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:23.166227 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.166214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvg8d\" (UniqueName: \"kubernetes.io/projected/866d9622-0c68-4e07-94ef-94cabbd34f86-kube-api-access-bvg8d\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:23.166893 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.166879 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 08:54:23.266782 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.266755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/866d9622-0c68-4e07-94ef-94cabbd34f86-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:23.267160 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.266793 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-wtmp\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.267160 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.266819 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-accelerators-collector-config\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.267160 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.266885 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.267160 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.266917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/866d9622-0c68-4e07-94ef-94cabbd34f86-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:23.267160 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.266964 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.267160 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.267000 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/866d9622-0c68-4e07-94ef-94cabbd34f86-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:23.267160 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.267050 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1ecaa283-05a2-4f68-857e-dd14c857c804-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.267160 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.267093 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.267160 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.267151 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-tls\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.267641 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.267206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-root\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.267641 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.267261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-sys\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.267641 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.267297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.267641 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.267316 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5tj\" (UniqueName: \"kubernetes.io/projected/6a763f73-62ad-43ed-babf-fb2ced67616a-kube-api-access-hn5tj\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.267641 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.267333 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l75cz\" (UniqueName: \"kubernetes.io/projected/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-api-access-l75cz\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.267641 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.267382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvg8d\" (UniqueName: \"kubernetes.io/projected/866d9622-0c68-4e07-94ef-94cabbd34f86-kube-api-access-bvg8d\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:23.267641 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.267581 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ecaa283-05a2-4f68-857e-dd14c857c804-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.267937 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.267672 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a763f73-62ad-43ed-babf-fb2ced67616a-metrics-client-ca\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.267937 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.267721 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-textfile\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.267937 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.267736 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/866d9622-0c68-4e07-94ef-94cabbd34f86-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:23.269252 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.269234 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/866d9622-0c68-4e07-94ef-94cabbd34f86-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:23.269402 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.269387 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/866d9622-0c68-4e07-94ef-94cabbd34f86-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:23.285896 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.285873 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvg8d\" (UniqueName: \"kubernetes.io/projected/866d9622-0c68-4e07-94ef-94cabbd34f86-kube-api-access-bvg8d\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:23.368029 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.368130 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368039 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5tj\" (UniqueName: \"kubernetes.io/projected/6a763f73-62ad-43ed-babf-fb2ced67616a-kube-api-access-hn5tj\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.368130 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l75cz\" (UniqueName: \"kubernetes.io/projected/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-api-access-l75cz\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.368198 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368158 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ecaa283-05a2-4f68-857e-dd14c857c804-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.368198 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368186 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a763f73-62ad-43ed-babf-fb2ced67616a-metrics-client-ca\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.368274 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-textfile\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.368274 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368234 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-wtmp\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.368385 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368367 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-wtmp\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.368433 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-accelerators-collector-config\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.368483 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368462 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.368541 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368490 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.368541 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368534 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1ecaa283-05a2-4f68-857e-dd14c857c804-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.368670 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368580 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.368670 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368621 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-tls\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.368670 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368644 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-textfile\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.368901 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368875 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a763f73-62ad-43ed-babf-fb2ced67616a-metrics-client-ca\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.369010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368649 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-root\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.369010 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.368967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-sys\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.369113 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.369012 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1ecaa283-05a2-4f68-857e-dd14c857c804-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.369113 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.369064 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ecaa283-05a2-4f68-857e-dd14c857c804-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.369225 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.369154 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-sys\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.369225 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.369159 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.369225 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.369202 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-root\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.369361 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.369249 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-accelerators-collector-config\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.370946 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.370908 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.371745 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.371716 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.371845 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.371809 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-tls\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.375436 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.375415 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.378185 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.378165 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:23.387023 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.387004 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5tj\" (UniqueName: \"kubernetes.io/projected/6a763f73-62ad-43ed-babf-fb2ced67616a-kube-api-access-hn5tj\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.387380 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.387361 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l75cz\" (UniqueName: \"kubernetes.io/projected/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-api-access-l75cz\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.426127 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.426100 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:23.454051 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.454024 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:23.465177 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:54:23.465151 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a763f73_62ad_43ed_babf_fb2ced67616a.slice/crio-a5edfa0961397ed24bc8093e9253a61ceb6ef2f303041c738a5e27ba30f0487b WatchSource:0}: Error finding container a5edfa0961397ed24bc8093e9253a61ceb6ef2f303041c738a5e27ba30f0487b: Status 404 returned error can't find the container with id a5edfa0961397ed24bc8093e9253a61ceb6ef2f303041c738a5e27ba30f0487b Oct 30 08:54:23.679552 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.679490 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2"] Oct 30 08:54:23.684570 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:54:23.684537 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866d9622_0c68_4e07_94ef_94cabbd34f86.slice/crio-9cfd9aa07e797b37a7411dda20860cbee1995353ef4e8a79b3df01affd76cb19 WatchSource:0}: Error finding container 9cfd9aa07e797b37a7411dda20860cbee1995353ef4e8a79b3df01affd76cb19: Status 404 returned error can't find the container with id 9cfd9aa07e797b37a7411dda20860cbee1995353ef4e8a79b3df01affd76cb19 Oct 30 08:54:23.721730 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:23.721708 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg"] Oct 30 08:54:23.724901 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:54:23.724879 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecaa283_05a2_4f68_857e_dd14c857c804.slice/crio-a1122495636b6dc03c9ece1042e2485fa558d1c4a26618100434e18a196d9587 WatchSource:0}: Error finding container a1122495636b6dc03c9ece1042e2485fa558d1c4a26618100434e18a196d9587: Status 404 returned error can't find the container with id a1122495636b6dc03c9ece1042e2485fa558d1c4a26618100434e18a196d9587 Oct 30 08:54:24.206906 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.206866 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" event={"ID":"1ecaa283-05a2-4f68-857e-dd14c857c804","Type":"ContainerStarted","Data":"a1122495636b6dc03c9ece1042e2485fa558d1c4a26618100434e18a196d9587"} Oct 30 08:54:24.209376 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.209340 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" event={"ID":"866d9622-0c68-4e07-94ef-94cabbd34f86","Type":"ContainerStarted","Data":"9009a1d0369c9c34d21602fe4b63c10a5f2140ea4d171c4ca6c32e35ed25a609"} Oct 30 08:54:24.209500 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.209400 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" event={"ID":"866d9622-0c68-4e07-94ef-94cabbd34f86","Type":"ContainerStarted","Data":"490277b0b44a2944aecda8632fc55848f03ac045986cb27bfb4b372c76175b0b"} Oct 30 08:54:24.209500 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.209430 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" event={"ID":"866d9622-0c68-4e07-94ef-94cabbd34f86","Type":"ContainerStarted","Data":"9cfd9aa07e797b37a7411dda20860cbee1995353ef4e8a79b3df01affd76cb19"} Oct 30 08:54:24.210807 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.210784 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8kgvx" event={"ID":"6a763f73-62ad-43ed-babf-fb2ced67616a","Type":"ContainerStarted","Data":"a5edfa0961397ed24bc8093e9253a61ceb6ef2f303041c738a5e27ba30f0487b"} Oct 30 08:54:24.418875 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.418849 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 08:54:24.422976 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.422953 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.440315 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.440281 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 08:54:24.440433 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.440344 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 08:54:24.440433 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.440369 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 08:54:24.440606 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.440452 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 08:54:24.440606 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.440479 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 08:54:24.440606 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.440491 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 08:54:24.440606 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.440519 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 08:54:24.440606 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.440566 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-752kq\"" Oct 30 08:54:24.443534 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.443514 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 08:54:24.443534 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.443529 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 08:54:24.452981 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.452903 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 08:54:24.579735 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.579701 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.579912 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.579752 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.579912 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.579792 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-out\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.579912 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.579901 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.580045 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.579926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.580045 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.579953 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.580045 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.579971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j75nb\" (UniqueName: \"kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-kube-api-access-j75nb\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.580045 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.580006 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-volume\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.580045 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.580033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.580219 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.580050 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.580219 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.580066 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.580219 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.580099 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-web-config\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.580219 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.580133 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.681394 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.681356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.681545 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.681412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.681545 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.681441 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j75nb\" (UniqueName: \"kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-kube-api-access-j75nb\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.681545 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.681490 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-volume\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.681545 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.681515 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.681545 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.681539 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.681755 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.681565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.681755 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.681627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-web-config\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.681755 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.681658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.681755 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.681683 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.681755 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.681712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.681755 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.681742 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-out\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.681927 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.681801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.681970 ip-10-0-135-148 kubenswrapper[2578]: E1030 08:54:24.681958 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-trusted-ca-bundle podName:b0ac551a-4850-4c58-9fe8-6432e5d3301c nodeName:}" failed. No retries permitted until 2026-10-30 08:54:25.181940609 +0000 UTC m=+54.935092203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "b0ac551a-4850-4c58-9fe8-6432e5d3301c") : configmap references non-existent config key: ca-bundle.crt Oct 30 08:54:24.684448 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.682803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.684448 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.683838 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.686579 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.686552 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-out\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.687423 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.687083 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.687423 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.687375 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.687573 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.687516 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.687778 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.687755 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-web-config\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.688223 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.688187 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-volume\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.689323 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.689302 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.690267 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.690247 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.690427 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.690400 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:24.701806 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:24.701786 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j75nb\" (UniqueName: \"kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-kube-api-access-j75nb\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:25.185657 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.185616 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:25.186600 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.186567 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:25.215546 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.215513 2578 generic.go:356] "Generic (PLEG): container finished" podID="6a763f73-62ad-43ed-babf-fb2ced67616a" containerID="befc92f6f4d381e6b9be57d97653b7eaed1d0d894d8425b86494a64f33c96cd2" exitCode=0 Oct 30 08:54:25.215704 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.215615 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8kgvx" event={"ID":"6a763f73-62ad-43ed-babf-fb2ced67616a","Type":"ContainerDied","Data":"befc92f6f4d381e6b9be57d97653b7eaed1d0d894d8425b86494a64f33c96cd2"} Oct 30 08:54:25.226622 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.221810 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" event={"ID":"1ecaa283-05a2-4f68-857e-dd14c857c804","Type":"ContainerStarted","Data":"0ca77a7e8958efdfc77cf3df70383a8c7e8764e49aefc206e18197a419bf945f"} Oct 30 08:54:25.226622 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.221873 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" event={"ID":"1ecaa283-05a2-4f68-857e-dd14c857c804","Type":"ContainerStarted","Data":"82eccf418569b8658fb499ee3439c6fdcbfb1391efd12c9817e15f134dd216f2"} Oct 30 08:54:25.226622 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.221903 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" event={"ID":"1ecaa283-05a2-4f68-857e-dd14c857c804","Type":"ContainerStarted","Data":"650bee986e9c04350f67ee153eb2cc9ef681063b15c2e0e2b5a2aff1946c72a2"} Oct 30 08:54:25.226622 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.225735 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" event={"ID":"866d9622-0c68-4e07-94ef-94cabbd34f86","Type":"ContainerStarted","Data":"bdfda7d013d8cbbfdc9ba50f67b09b80a2dfa3ac2e7f2cec2e5c0ab083d1697a"} Oct 30 08:54:25.254802 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.254778 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx"] Oct 30 08:54:25.260053 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.260032 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.275146 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.275127 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 08:54:25.275259 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.275236 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4gqanurtnecfq\"" Oct 30 08:54:25.275259 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.275159 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 08:54:25.275430 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.275159 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 08:54:25.275430 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.275269 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qz7rt\"" Oct 30 08:54:25.275430 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.275176 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 08:54:25.275430 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.275129 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 08:54:25.279747 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.279728 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx"] Oct 30 08:54:25.298808 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.298759 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" podStartSLOduration=1.086765176 podStartE2EDuration="2.298740887s" totalImagesPullingTime="1.211975711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:25.297801162 +0000 UTC m=+55.050952767" watchObservedRunningTime="2026-10-30 08:54:25.298740887 +0000 UTC m=+55.051892476" Oct 30 08:54:25.336702 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.336652 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" podStartSLOduration=1.294473944 podStartE2EDuration="2.336637518s" totalImagesPullingTime="1.042163574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:25.335561116 +0000 UTC m=+55.088712706" watchObservedRunningTime="2026-10-30 08:54:25.336637518 +0000 UTC m=+55.089789106" Oct 30 08:54:25.356238 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.356217 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:25.387789 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.387766 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-metrics-client-ca\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.387909 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.387833 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.387944 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.387916 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.387944 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.387937 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-grpc-tls\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.388019 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.387969 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9984j\" (UniqueName: \"kubernetes.io/projected/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-kube-api-access-9984j\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.388019 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.387988 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.388019 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.388003 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-tmp\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.388103 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.388035 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.388103 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.388076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-tls\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.488809 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.488729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9984j\" (UniqueName: \"kubernetes.io/projected/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-kube-api-access-9984j\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.488809 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.488773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.488809 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.488801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-tmp\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.489353 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.488833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.489353 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.488869 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-tls\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.489353 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.489045 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-metrics-client-ca\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.489353 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.489105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.489353 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.489236 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-tmp\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.489353 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.489283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.489353 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.489322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-grpc-tls\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.490079 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.490023 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-metrics-client-ca\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.491632 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.491607 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.491836 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.491816 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.492087 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.492069 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-tls\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.492179 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.492162 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.492417 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.492391 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-grpc-tls\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.492543 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.492518 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.507969 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.507940 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9984j\" (UniqueName: \"kubernetes.io/projected/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-kube-api-access-9984j\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.602210 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.602177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:25.686852 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.686798 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 08:54:25.690963 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:54:25.690934 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ac551a_4850_4c58_9fe8_6432e5d3301c.slice/crio-f6ee89a073f2369082eb1f51a558980516571acf01d445de228953a6a7804334 WatchSource:0}: Error finding container f6ee89a073f2369082eb1f51a558980516571acf01d445de228953a6a7804334: Status 404 returned error can't find the container with id f6ee89a073f2369082eb1f51a558980516571acf01d445de228953a6a7804334 Oct 30 08:54:25.899760 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:25.899733 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx"] Oct 30 08:54:25.901570 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:54:25.901548 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5d7c66_ce15_4d20_bc37_4d832f308d9f.slice/crio-7ca0a097a365338c29c4ce0db210c8da5e34c1079d3c0a8264abf33f3c055ca0 WatchSource:0}: Error finding container 7ca0a097a365338c29c4ce0db210c8da5e34c1079d3c0a8264abf33f3c055ca0: Status 404 returned error can't find the container with id 7ca0a097a365338c29c4ce0db210c8da5e34c1079d3c0a8264abf33f3c055ca0 Oct 30 08:54:26.230991 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:26.230853 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" event={"ID":"7c5d7c66-ce15-4d20-bc37-4d832f308d9f","Type":"ContainerStarted","Data":"7ca0a097a365338c29c4ce0db210c8da5e34c1079d3c0a8264abf33f3c055ca0"} Oct 30 08:54:26.233505 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:26.233472 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8kgvx" event={"ID":"6a763f73-62ad-43ed-babf-fb2ced67616a","Type":"ContainerStarted","Data":"e5744e60304983ab4b029b8fa4a7d273da28a442b52839667224350ed924bcbc"} Oct 30 08:54:26.233723 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:26.233528 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8kgvx" event={"ID":"6a763f73-62ad-43ed-babf-fb2ced67616a","Type":"ContainerStarted","Data":"b8bb4d7385723a34b4c4c1e895d69ba33afb63eaa118ec3a8bbd800debe773a5"} Oct 30 08:54:26.235066 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:26.234988 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0ac551a-4850-4c58-9fe8-6432e5d3301c","Type":"ContainerStarted","Data":"f6ee89a073f2369082eb1f51a558980516571acf01d445de228953a6a7804334"} Oct 30 08:54:26.276235 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:26.276178 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8kgvx" podStartSLOduration=2.309824283 podStartE2EDuration="3.276158624s" totalImagesPullingTime="878.010668ms" totalInitContainerRuntime="88.323673ms" isStatefulPod=true podCreationTimestamp="2026-10-30 08:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:26.275625443 +0000 UTC m=+56.028777035" watchObservedRunningTime="2026-10-30 08:54:26.276158624 +0000 UTC m=+56.029310216" Oct 30 08:54:27.239897 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:27.239865 2578 generic.go:356] "Generic (PLEG): container finished" podID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerID="428a41797386d649baf498c2ec11969043e66d787e4e260eb3d334cd7fc51391" exitCode=0 Oct 30 08:54:27.240361 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:27.239959 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0ac551a-4850-4c58-9fe8-6432e5d3301c","Type":"ContainerDied","Data":"428a41797386d649baf498c2ec11969043e66d787e4e260eb3d334cd7fc51391"} Oct 30 08:54:28.245546 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.245518 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" event={"ID":"7c5d7c66-ce15-4d20-bc37-4d832f308d9f","Type":"ContainerStarted","Data":"3240d534699292f1eafed6474fc6407151d800846e229f9d56417f27cfdd470c"} Oct 30 08:54:28.245944 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.245581 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" event={"ID":"7c5d7c66-ce15-4d20-bc37-4d832f308d9f","Type":"ContainerStarted","Data":"80b9aa7a5010138b4c76ea67962a24785045c2997e115d0f7be08286e480eb92"} Oct 30 08:54:28.245944 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.245621 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" event={"ID":"7c5d7c66-ce15-4d20-bc37-4d832f308d9f","Type":"ContainerStarted","Data":"f9cc724657029b2991639a62bc7405de02b2ad9008f6388b383d0fdde40b1102"} Oct 30 08:54:28.427917 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.427887 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-65c85d64c4-d8zwz"] Oct 30 08:54:28.431649 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.431624 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.447232 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.447197 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65c85d64c4-d8zwz"] Oct 30 08:54:28.457836 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.457806 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 08:54:28.457935 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.457867 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 08:54:28.457935 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.457896 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4xr7r\"" Oct 30 08:54:28.457935 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.457876 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 08:54:28.458095 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.457965 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 08:54:28.458220 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.458197 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 08:54:28.472086 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.472052 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 08:54:28.620091 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.619690 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-federate-client-tls\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.620091 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.619741 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.620091 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.619777 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-telemeter-client-tls\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.620091 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.619799 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-secret-telemeter-client\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.620091 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.619820 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.620091 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.619860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c94rp\" (UniqueName: \"kubernetes.io/projected/3e6de072-eb99-48f0-ade6-f7fad4be241b-kube-api-access-c94rp\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.620091 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.619904 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-serving-certs-ca-bundle\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.620091 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.619954 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-metrics-client-ca\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.720330 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.720294 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-federate-client-tls\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.720518 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.720338 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.720518 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.720492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-telemeter-client-tls\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.720671 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.720538 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-secret-telemeter-client\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.720671 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.720578 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.720671 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.720663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c94rp\" (UniqueName: \"kubernetes.io/projected/3e6de072-eb99-48f0-ade6-f7fad4be241b-kube-api-access-c94rp\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.720815 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.720715 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-serving-certs-ca-bundle\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.720815 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.720754 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-metrics-client-ca\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.721358 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.721326 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.721762 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.721735 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-metrics-client-ca\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.721882 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.721855 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-serving-certs-ca-bundle\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.723457 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.723434 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-federate-client-tls\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.724128 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.724082 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-secret-telemeter-client\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.724221 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.724158 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.724611 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.724574 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-telemeter-client-tls\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.740148 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.740126 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c94rp\" (UniqueName: \"kubernetes.io/projected/3e6de072-eb99-48f0-ade6-f7fad4be241b-kube-api-access-c94rp\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:28.753887 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:28.753869 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:29.128456 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.128376 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65c85d64c4-d8zwz"] Oct 30 08:54:29.133053 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:54:29.133021 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6de072_eb99_48f0_ade6_f7fad4be241b.slice/crio-cff2c8fb73b97f43e415a94fbbf0774aca7c04bde33f4c8e0337d8ee65ecaaf3 WatchSource:0}: Error finding container cff2c8fb73b97f43e415a94fbbf0774aca7c04bde33f4c8e0337d8ee65ecaaf3: Status 404 returned error can't find the container with id cff2c8fb73b97f43e415a94fbbf0774aca7c04bde33f4c8e0337d8ee65ecaaf3 Oct 30 08:54:29.251633 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.251605 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0ac551a-4850-4c58-9fe8-6432e5d3301c","Type":"ContainerStarted","Data":"e7945184e2389a6ef4ffb71f3ac48af885e17a22340468496b63ca8ae5ab1dde"} Oct 30 08:54:29.252003 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.251672 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0ac551a-4850-4c58-9fe8-6432e5d3301c","Type":"ContainerStarted","Data":"e95aa4a98351e9569f8478ccac710143820e3866cd01b81f1aecb8a9f09eb8a4"} Oct 30 08:54:29.252003 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.251741 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0ac551a-4850-4c58-9fe8-6432e5d3301c","Type":"ContainerStarted","Data":"e586f4097d1677601287d5f8ab7023f50417fe891c9c2b6f85eae471ce880aec"} Oct 30 08:54:29.252003 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.251766 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0ac551a-4850-4c58-9fe8-6432e5d3301c","Type":"ContainerStarted","Data":"16d53eaf6744f3c5571afdd113bfb07aea1a8e4d0fe80f1df510e6b77e0c168d"} Oct 30 08:54:29.252003 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.251795 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0ac551a-4850-4c58-9fe8-6432e5d3301c","Type":"ContainerStarted","Data":"fdf66999accbbe1ad4c116bb62c82a6764715ed865aa7c9091575836470a4434"} Oct 30 08:54:29.252003 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.251823 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0ac551a-4850-4c58-9fe8-6432e5d3301c","Type":"ContainerStarted","Data":"827272fdb9b7e83447fd6d35d96c4726bc1f60d78fd3d96158aacabc56c387f2"} Oct 30 08:54:29.254686 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.254657 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" event={"ID":"7c5d7c66-ce15-4d20-bc37-4d832f308d9f","Type":"ContainerStarted","Data":"ec60f69a19245706bc7136bd58c534823ba47d509e22bfd2638881b2a9d43460"} Oct 30 08:54:29.254785 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.254735 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" event={"ID":"7c5d7c66-ce15-4d20-bc37-4d832f308d9f","Type":"ContainerStarted","Data":"ab6d496040842df31af60e2c7fd941ff09fe943921b4f854a50a0c49461578be"} Oct 30 08:54:29.254785 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.254781 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:29.254864 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.254810 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" event={"ID":"7c5d7c66-ce15-4d20-bc37-4d832f308d9f","Type":"ContainerStarted","Data":"a493f1700c5ce8a6e5647343b37b7db465a3246009d27c129067bd320e5a7ec9"} Oct 30 08:54:29.255749 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.255728 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" event={"ID":"3e6de072-eb99-48f0-ade6-f7fad4be241b","Type":"ContainerStarted","Data":"cff2c8fb73b97f43e415a94fbbf0774aca7c04bde33f4c8e0337d8ee65ecaaf3"} Oct 30 08:54:29.299399 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.299346 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.675694167 podStartE2EDuration="5.299329432s" totalImagesPullingTime="2.610400626s" totalInitContainerRuntime="13.234639ms" isStatefulPod=false podCreationTimestamp="2026-10-30 08:54:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:29.297514481 +0000 UTC m=+59.050666073" watchObservedRunningTime="2026-10-30 08:54:29.299329432 +0000 UTC m=+59.052481023" Oct 30 08:54:29.373107 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.373053 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" podStartSLOduration=1.663429542 podStartE2EDuration="4.373034808s" totalImagesPullingTime="2.709605266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:54:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:29.372088013 +0000 UTC m=+59.125239604" watchObservedRunningTime="2026-10-30 08:54:29.373034808 +0000 UTC m=+59.126186400" Oct 30 08:54:29.934293 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.934256 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 08:54:29.939294 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.939269 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:29.954864 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.954824 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 08:54:29.954996 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.954863 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 08:54:29.955116 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.955097 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 08:54:29.955116 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.955107 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3a6is943ja23r\"" Oct 30 08:54:29.956977 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.955120 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 08:54:29.956977 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.955166 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 08:54:29.956977 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.955198 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 08:54:29.966761 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.966737 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5ptj4\"" Oct 30 08:54:29.968777 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.968756 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 08:54:29.972763 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.972738 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 08:54:29.972871 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.972835 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 08:54:29.986455 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.986420 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 08:54:29.993365 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.993333 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 08:54:29.999293 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:29.999267 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 08:54:30.035223 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035364 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035234 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035364 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035364 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035325 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035541 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035444 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-web-config\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035541 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config-out\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035541 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035517 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035743 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035562 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035743 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035646 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035743 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035743 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035710 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035743 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035738 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035984 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035768 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035984 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035806 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035984 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035844 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx5wx\" (UniqueName: \"kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-kube-api-access-tx5wx\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035984 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035887 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035984 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035913 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.035984 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.036263 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.035992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.036263 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.036020 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.055009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.054940 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 08:54:30.055009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.054963 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 08:54:30.055009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.054944 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 08:54:30.055009 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.054999 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-web-config\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137288 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config-out\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137329 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137461 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137538 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx5wx\" (UniqueName: \"kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-kube-api-access-tx5wx\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137721 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137745 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137769 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.138245 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.139160 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137843 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.139160 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.139160 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.137900 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.139160 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.138017 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.139160 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.138282 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.139160 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.138964 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.140921 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.140474 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.140921 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.140506 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.142774 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.141117 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.142774 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.141153 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-web-config\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.142774 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.141736 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.142774 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.141938 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.142774 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.142021 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.143513 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.143475 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.143896 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.143873 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.144118 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.144092 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.144246 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.144149 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.144500 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.144476 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.145095 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.145073 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config-out\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.145200 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.145173 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.146625 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.146607 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.146935 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.146912 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.158601 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.158563 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx5wx\" (UniqueName: \"kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-kube-api-access-tx5wx\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.257254 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.257193 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:30.276640 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.274979 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:30.620039 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:30.619912 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 08:54:30.636614 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:54:30.636562 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f15807d_cf9f_435a_84e6_febce2ef9ce4.slice/crio-de8e1fa5a3d2bd96f1b430825b435f48cf9617c4ea7431323c387f9d1daddfe2 WatchSource:0}: Error finding container de8e1fa5a3d2bd96f1b430825b435f48cf9617c4ea7431323c387f9d1daddfe2: Status 404 returned error can't find the container with id de8e1fa5a3d2bd96f1b430825b435f48cf9617c4ea7431323c387f9d1daddfe2 Oct 30 08:54:31.199794 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:31.199721 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:31.271105 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:31.271074 2578 generic.go:356] "Generic (PLEG): container finished" podID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerID="c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86" exitCode=0 Oct 30 08:54:31.271506 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:31.271169 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f15807d-cf9f-435a-84e6-febce2ef9ce4","Type":"ContainerDied","Data":"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86"} Oct 30 08:54:31.271506 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:31.271249 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f15807d-cf9f-435a-84e6-febce2ef9ce4","Type":"ContainerStarted","Data":"de8e1fa5a3d2bd96f1b430825b435f48cf9617c4ea7431323c387f9d1daddfe2"} Oct 30 08:54:31.273817 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:31.273775 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" event={"ID":"3e6de072-eb99-48f0-ade6-f7fad4be241b","Type":"ContainerStarted","Data":"b60d4d183a721d433097ce2780dc0fbcdbfa1484ba1d091bdda6695c08208c6a"} Oct 30 08:54:31.273868 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:31.273827 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" event={"ID":"3e6de072-eb99-48f0-ade6-f7fad4be241b","Type":"ContainerStarted","Data":"41e60db03c7d552a3fa3dc8ee9389d2d55014f368f828462d8d7e7c3efa638c8"} Oct 30 08:54:31.273868 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:31.273863 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" event={"ID":"3e6de072-eb99-48f0-ade6-f7fad4be241b","Type":"ContainerStarted","Data":"f435b8ab5de94a5a461b5823aadb08bdd5e09b27451182c8d7866e29c04670ce"} Oct 30 08:54:31.360638 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:31.360573 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" podStartSLOduration=1.5968846330000002 podStartE2EDuration="3.360559257s" totalImagesPullingTime="1.763674624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:54:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:31.359494546 +0000 UTC m=+61.112646139" watchObservedRunningTime="2026-10-30 08:54:31.360559257 +0000 UTC m=+61.113710848" Oct 30 08:54:35.267290 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:35.267264 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:35.295204 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:35.295166 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f15807d-cf9f-435a-84e6-febce2ef9ce4","Type":"ContainerStarted","Data":"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a"} Oct 30 08:54:35.295323 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:35.295245 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f15807d-cf9f-435a-84e6-febce2ef9ce4","Type":"ContainerStarted","Data":"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e"} Oct 30 08:54:35.295323 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:35.295283 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f15807d-cf9f-435a-84e6-febce2ef9ce4","Type":"ContainerStarted","Data":"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1"} Oct 30 08:54:35.295410 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:35.295323 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f15807d-cf9f-435a-84e6-febce2ef9ce4","Type":"ContainerStarted","Data":"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119"} Oct 30 08:54:35.295410 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:35.295364 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f15807d-cf9f-435a-84e6-febce2ef9ce4","Type":"ContainerStarted","Data":"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661"} Oct 30 08:54:35.295410 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:35.295392 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f15807d-cf9f-435a-84e6-febce2ef9ce4","Type":"ContainerStarted","Data":"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c"} Oct 30 08:54:35.374072 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:35.374028 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.10671657 podStartE2EDuration="6.374013872s" totalImagesPullingTime="3.239910971s" totalInitContainerRuntime="27.386331ms" isStatefulPod=false podCreationTimestamp="2026-10-30 08:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:35.373476731 +0000 UTC m=+65.126628356" watchObservedRunningTime="2026-10-30 08:54:35.374013872 +0000 UTC m=+65.127165463" Oct 30 08:54:36.701166 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:36.701129 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:36.701625 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:36.701190 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrjj\" (UniqueName: \"kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj\") pod \"network-check-target-tf9r8\" (UID: \"25cb63c1-5df3-4be8-b407-b9bf2641cf95\") " pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:36.701625 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:36.701226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:36.714965 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:36.714935 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 08:54:36.727333 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:36.727310 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 08:54:36.727333 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:36.727329 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 08:54:36.733832 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:36.733808 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:36.733915 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:36.733844 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:36.735415 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:36.735402 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 08:54:36.744729 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:36.744709 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szrjj\" (UniqueName: \"kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj\") pod \"network-check-target-tf9r8\" (UID: \"25cb63c1-5df3-4be8-b407-b9bf2641cf95\") " pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:36.833103 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:36.833077 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kgscj\"" Oct 30 08:54:36.840343 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:36.840321 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:36.845558 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:36.845540 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r2tvf\"" Oct 30 08:54:36.850647 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:36.850626 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7pnl7\"" Oct 30 08:54:36.852721 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:36.852698 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:36.857502 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:36.857481 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:37.140306 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:37.140280 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tf9r8"] Oct 30 08:54:37.144771 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:54:37.144744 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cb63c1_5df3_4be8_b407_b9bf2641cf95.slice/crio-7d403b9188ea911ab69bfac4f21afc5f7177a9afcb5bb35b6d93677935e996cf WatchSource:0}: Error finding container 7d403b9188ea911ab69bfac4f21afc5f7177a9afcb5bb35b6d93677935e996cf: Status 404 returned error can't find the container with id 7d403b9188ea911ab69bfac4f21afc5f7177a9afcb5bb35b6d93677935e996cf Oct 30 08:54:37.175429 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:37.175405 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m5sz6"] Oct 30 08:54:37.178520 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:37.178499 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ktt7j"] Oct 30 08:54:37.180190 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:54:37.180111 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7400965_54de_4420_a62a_37baf38d949d.slice/crio-a5ce646bfb584a3f1f9cbc2a1586240c601974f001775e87d087fcef54c40442 WatchSource:0}: Error finding container a5ce646bfb584a3f1f9cbc2a1586240c601974f001775e87d087fcef54c40442: Status 404 returned error can't find the container with id a5ce646bfb584a3f1f9cbc2a1586240c601974f001775e87d087fcef54c40442 Oct 30 08:54:37.182074 ip-10-0-135-148 kubenswrapper[2578]: W1030 08:54:37.182050 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82405fd_bde5_4428_8b23_2be99042feef.slice/crio-de77cd9d9fb7cca1b8e6906c237d1a8e485707520566a42a75f7953b47f75707 WatchSource:0}: Error finding container de77cd9d9fb7cca1b8e6906c237d1a8e485707520566a42a75f7953b47f75707: Status 404 returned error can't find the container with id de77cd9d9fb7cca1b8e6906c237d1a8e485707520566a42a75f7953b47f75707 Oct 30 08:54:37.305298 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:37.305261 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m5sz6" event={"ID":"c7400965-54de-4420-a62a-37baf38d949d","Type":"ContainerStarted","Data":"a5ce646bfb584a3f1f9cbc2a1586240c601974f001775e87d087fcef54c40442"} Oct 30 08:54:37.306500 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:37.306468 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tf9r8" event={"ID":"25cb63c1-5df3-4be8-b407-b9bf2641cf95","Type":"ContainerStarted","Data":"7d403b9188ea911ab69bfac4f21afc5f7177a9afcb5bb35b6d93677935e996cf"} Oct 30 08:54:37.307574 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:37.307553 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ktt7j" event={"ID":"f82405fd-bde5-4428-8b23-2be99042feef","Type":"ContainerStarted","Data":"de77cd9d9fb7cca1b8e6906c237d1a8e485707520566a42a75f7953b47f75707"} Oct 30 08:54:39.320994 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:39.320955 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ktt7j" event={"ID":"f82405fd-bde5-4428-8b23-2be99042feef","Type":"ContainerStarted","Data":"463cf1c8a53abeb0a4350519bc21f96b7357bcc2d39a6b4b2778497c9ffca44f"} Oct 30 08:54:39.321406 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:39.321016 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ktt7j" event={"ID":"f82405fd-bde5-4428-8b23-2be99042feef","Type":"ContainerStarted","Data":"0446140bd984a47847d4739c3d622889a216781105011ed0bc9a52f4bb3905dd"} Oct 30 08:54:39.360084 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:39.358289 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ktt7j" podStartSLOduration=67.252928972 podStartE2EDuration="1m8.358271976s" totalImagesPullingTime="1.105343004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:39.358149505 +0000 UTC m=+69.111301099" watchObservedRunningTime="2026-10-30 08:54:39.358271976 +0000 UTC m=+69.111423570" Oct 30 08:54:40.278624 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:40.277369 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:41.334471 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:41.334431 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tf9r8" event={"ID":"25cb63c1-5df3-4be8-b407-b9bf2641cf95","Type":"ContainerStarted","Data":"54dbee535405cbdff78b15856ad918b88d225787cbf80edbe68a6e9051b2f75d"} Oct 30 08:54:41.334932 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:41.334739 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:41.375335 ip-10-0-135-148 kubenswrapper[2578]: I1030 08:54:41.375260 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tf9r8" podStartSLOduration=67.128263899 podStartE2EDuration="1m10.375242436s" totalImagesPullingTime="3.246978537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:41.372462915 +0000 UTC m=+71.125614507" watchObservedRunningTime="2026-10-30 08:54:41.375242436 +0000 UTC m=+71.128394027" Oct 30 08:54:42.079370 ip-10-0-135-148 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 08:54:42.091958 ip-10-0-135-148 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 08:54:42.092134 ip-10-0-135-148 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 08:54:42.092185 ip-10-0-135-148 systemd[1]: kubelet.service: Consumed 4.352s CPU time. Oct 30 08:54:42.105186 ip-10-0-135-148 systemd[1]: Starting Kubernetes Kubelet... Oct 30 08:54:42.168051 ip-10-0-135-148 kubenswrapper[5938]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:54:42.168051 ip-10-0-135-148 kubenswrapper[5938]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 08:54:42.168051 ip-10-0-135-148 kubenswrapper[5938]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:54:42.168051 ip-10-0-135-148 kubenswrapper[5938]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:54:42.171011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.170836 5938 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171012 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171021 5938 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171025 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171030 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171034 5938 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171039 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171044 5938 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171048 5938 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171052 5938 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171055 5938 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171061 5938 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171065 5938 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171069 5938 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171073 5938 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171077 5938 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171081 5938 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171085 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171089 5938 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171093 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:54:42.171110 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171097 5938 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171101 5938 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171105 5938 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171109 5938 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171114 5938 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171119 5938 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171123 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171127 5938 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171132 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171135 5938 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171140 5938 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171144 5938 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171147 5938 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171152 5938 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171156 5938 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171161 5938 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171166 5938 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171170 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171174 5938 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171179 5938 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:54:42.172008 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171183 5938 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171188 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171192 5938 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171195 5938 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171199 5938 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171204 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171210 5938 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171215 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171219 5938 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171223 5938 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171228 5938 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171232 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171236 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171240 5938 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171245 5938 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171249 5938 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171254 5938 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171258 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171262 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171267 5938 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:54:42.172895 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171272 5938 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171276 5938 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171280 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171290 5938 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171298 5938 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171305 5938 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171311 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171316 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171320 5938 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171324 5938 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171329 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171333 5938 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171337 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171341 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171345 5938 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171349 5938 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171353 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171358 5938 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171362 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:54:42.173805 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171366 5938 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171371 5938 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171375 5938 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171380 5938 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171384 5938 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171388 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171392 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171397 5938 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171401 5938 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171406 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171410 5938 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171414 5938 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171418 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171422 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171427 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171431 5938 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171435 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171440 5938 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171444 5938 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:54:42.174681 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171448 5938 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171452 5938 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171459 5938 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171463 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171467 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171472 5938 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171476 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.171480 5938 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172794 5938 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172805 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172812 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172818 5938 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172823 5938 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172828 5938 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172833 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172837 5938 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172841 5938 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172845 5938 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172850 5938 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172855 5938 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:54:42.175528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172860 5938 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172864 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172869 5938 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172873 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172877 5938 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172882 5938 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172886 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172890 5938 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172894 5938 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172898 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172902 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172906 5938 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172910 5938 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172914 5938 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172918 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172922 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172926 5938 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172931 5938 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172935 5938 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172939 5938 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:54:42.176446 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172946 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172950 5938 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172954 5938 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172958 5938 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172963 5938 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172967 5938 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172971 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172975 5938 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172981 5938 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172988 5938 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172993 5938 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.172997 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173002 5938 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173007 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173011 5938 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173015 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173020 5938 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173024 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173028 5938 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:54:42.177309 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173032 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173036 5938 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173040 5938 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173045 5938 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173049 5938 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173053 5938 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173057 5938 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173065 5938 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173070 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173075 5938 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173080 5938 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173085 5938 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173089 5938 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173095 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173100 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173104 5938 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173108 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173113 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173117 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:54:42.177870 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173121 5938 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173126 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173130 5938 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173134 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173138 5938 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173142 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173147 5938 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173152 5938 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173156 5938 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173160 5938 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173164 5938 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173168 5938 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173173 5938 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173177 5938 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173181 5938 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173185 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173189 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173193 5938 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173198 5938 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173203 5938 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:54:42.178338 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173207 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173211 5938 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173215 5938 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173219 5938 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173223 5938 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173227 5938 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173233 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173237 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173242 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173246 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173250 5938 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173254 5938 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173261 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173265 5938 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173269 5938 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.173273 5938 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174077 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174088 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174094 5938 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:54:42.178845 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174099 5938 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174103 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174108 5938 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174112 5938 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174116 5938 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174122 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174126 5938 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174131 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174135 5938 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174139 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174143 5938 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174147 5938 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174152 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174156 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174160 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174164 5938 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174168 5938 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174172 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174177 5938 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174182 5938 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:54:42.179390 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174186 5938 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174190 5938 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174195 5938 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174199 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174203 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174207 5938 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174213 5938 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174220 5938 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174225 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174229 5938 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174233 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174238 5938 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174243 5938 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174247 5938 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174253 5938 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174257 5938 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174261 5938 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174265 5938 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174270 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174274 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:54:42.179919 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174278 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174282 5938 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174286 5938 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174290 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174295 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174299 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174303 5938 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174307 5938 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174311 5938 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174315 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174319 5938 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174324 5938 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174328 5938 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174332 5938 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174336 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174340 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174345 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174349 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174353 5938 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174357 5938 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:54:42.180403 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174361 5938 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174366 5938 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174370 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174374 5938 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174378 5938 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174383 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174387 5938 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174391 5938 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174396 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174403 5938 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174410 5938 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174415 5938 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174419 5938 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174423 5938 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174428 5938 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174432 5938 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174435 5938 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174440 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174444 5938 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:54:42.180897 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174449 5938 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174453 5938 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174457 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174462 5938 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174466 5938 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174471 5938 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174475 5938 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174479 5938 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174484 5938 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174488 5938 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174492 5938 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174496 5938 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174500 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174504 5938 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174508 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174513 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174517 5938 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174522 5938 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174526 5938 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:54:42.181354 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174532 5938 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174536 5938 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174541 5938 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174545 5938 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.174549 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174689 5938 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174707 5938 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174717 5938 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174724 5938 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174730 5938 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174735 5938 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174741 5938 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174748 5938 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174753 5938 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174758 5938 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174763 5938 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174769 5938 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174774 5938 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174778 5938 flags.go:64] FLAG: --cgroup-root="" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174783 5938 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174787 5938 flags.go:64] FLAG: --client-ca-file="" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174791 5938 flags.go:64] FLAG: --cloud-provider="external" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174796 5938 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174802 5938 flags.go:64] FLAG: --cluster-domain="" Oct 30 08:54:42.181859 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174806 5938 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174811 5938 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174816 5938 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174821 5938 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174827 5938 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174831 5938 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174836 5938 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174841 5938 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174845 5938 flags.go:64] FLAG: --contention-profiling="false" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174853 5938 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174858 5938 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174863 5938 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174868 5938 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174874 5938 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174879 5938 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174883 5938 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174890 5938 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174895 5938 flags.go:64] FLAG: --enable-server="true" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174899 5938 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174906 5938 flags.go:64] FLAG: --event-burst="100" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174911 5938 flags.go:64] FLAG: --event-qps="50" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174915 5938 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174920 5938 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174925 5938 flags.go:64] FLAG: --eviction-hard="" Oct 30 08:54:42.182432 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174931 5938 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174935 5938 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174941 5938 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174945 5938 flags.go:64] FLAG: --eviction-soft="" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174950 5938 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174955 5938 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174960 5938 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174964 5938 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174968 5938 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174973 5938 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174978 5938 flags.go:64] FLAG: --feature-gates="" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174984 5938 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174989 5938 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.174994 5938 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175000 5938 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175004 5938 flags.go:64] FLAG: --healthz-port="10248" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175009 5938 flags.go:64] FLAG: --help="false" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175019 5938 flags.go:64] FLAG: --hostname-override="ip-10-0-135-148.ec2.internal" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175025 5938 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175030 5938 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175034 5938 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175040 5938 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175045 5938 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175050 5938 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 08:54:42.183043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175056 5938 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175061 5938 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175066 5938 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175071 5938 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175076 5938 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175080 5938 flags.go:64] FLAG: --kube-reserved="" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175085 5938 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175089 5938 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175094 5938 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175099 5938 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175103 5938 flags.go:64] FLAG: --lock-file="" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175107 5938 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175111 5938 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175116 5938 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175125 5938 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175130 5938 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175134 5938 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175139 5938 flags.go:64] FLAG: --logging-format="text" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175143 5938 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175148 5938 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175153 5938 flags.go:64] FLAG: --manifest-url="" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175157 5938 flags.go:64] FLAG: --manifest-url-header="" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175164 5938 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175168 5938 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175174 5938 flags.go:64] FLAG: --max-pods="110" Oct 30 08:54:42.183643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175181 5938 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175186 5938 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175190 5938 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175195 5938 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175200 5938 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175205 5938 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175209 5938 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175222 5938 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175226 5938 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175231 5938 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175236 5938 flags.go:64] FLAG: --pod-cidr="" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175241 5938 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175245 5938 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175250 5938 flags.go:64] FLAG: --pods-per-core="0" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175254 5938 flags.go:64] FLAG: --port="10250" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175259 5938 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175264 5938 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01fa2920a76d3fbe5" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175268 5938 flags.go:64] FLAG: --qos-reserved="" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175273 5938 flags.go:64] FLAG: --read-only-port="10255" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175278 5938 flags.go:64] FLAG: --register-node="true" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175282 5938 flags.go:64] FLAG: --register-with-taints="" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175288 5938 flags.go:64] FLAG: --registry-burst="10" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175293 5938 flags.go:64] FLAG: --registry-qps="5" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175297 5938 flags.go:64] FLAG: --reserved-cpus="" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175302 5938 flags.go:64] FLAG: --reserved-memory="" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175308 5938 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 08:54:42.184283 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175313 5938 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175317 5938 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175321 5938 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175326 5938 flags.go:64] FLAG: --runonce="false" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175331 5938 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175336 5938 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175342 5938 flags.go:64] FLAG: --seccomp-default="false" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175346 5938 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175351 5938 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175356 5938 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175361 5938 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175367 5938 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175371 5938 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175378 5938 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175383 5938 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175387 5938 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175392 5938 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175397 5938 flags.go:64] FLAG: --system-cgroups="" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175401 5938 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175409 5938 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175437 5938 flags.go:64] FLAG: --tls-cert-file="" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175443 5938 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175450 5938 flags.go:64] FLAG: --tls-min-version="" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175455 5938 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175459 5938 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 08:54:42.184986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175464 5938 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175468 5938 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175473 5938 flags.go:64] FLAG: --v="2" Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175479 5938 flags.go:64] FLAG: --version="false" Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175486 5938 flags.go:64] FLAG: --vmodule="" Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175497 5938 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.175502 5938 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175745 5938 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175755 5938 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175760 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175765 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175770 5938 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175774 5938 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175780 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175787 5938 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175793 5938 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175798 5938 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175803 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175808 5938 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175812 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:54:42.185757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175819 5938 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175823 5938 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175827 5938 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175832 5938 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175836 5938 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175840 5938 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175844 5938 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175849 5938 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175853 5938 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175858 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175862 5938 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175866 5938 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175870 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175875 5938 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175879 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175883 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175887 5938 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175892 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175896 5938 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:54:42.186263 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175900 5938 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175905 5938 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175909 5938 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175913 5938 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175917 5938 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175921 5938 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175927 5938 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175931 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175936 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175940 5938 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175944 5938 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175950 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175955 5938 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175960 5938 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175964 5938 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175968 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175972 5938 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175977 5938 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175982 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:54:42.187034 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175986 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175990 5938 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175994 5938 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.175998 5938 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176002 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176007 5938 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176011 5938 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176016 5938 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176020 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176024 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176028 5938 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176032 5938 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176037 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176041 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176045 5938 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176049 5938 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176053 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176058 5938 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176062 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176068 5938 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:54:42.187916 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176072 5938 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176077 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176081 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176085 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176089 5938 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176094 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176100 5938 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176104 5938 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176108 5938 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176112 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176116 5938 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176121 5938 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176125 5938 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176130 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176134 5938 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176138 5938 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176142 5938 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176146 5938 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176151 5938 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176155 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:54:42.188572 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176159 5938 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:54:42.189206 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176163 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:54:42.189206 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176167 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:54:42.189206 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176172 5938 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:54:42.189206 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176176 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:54:42.189206 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176180 5938 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:54:42.189206 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176184 5938 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:54:42.189206 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176190 5938 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:54:42.189206 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176196 5938 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:54:42.189206 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176201 5938 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:54:42.189206 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176205 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:54:42.189206 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176211 5938 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:54:42.189206 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176215 5938 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:54:42.189206 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176219 5938 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:54:42.189206 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176223 5938 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:54:42.189206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.176265 5938 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:54:42.189206 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176338 5938 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176345 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176351 5938 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176357 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176361 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176366 5938 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176370 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176374 5938 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176379 5938 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176383 5938 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176387 5938 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176391 5938 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176395 5938 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176399 5938 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176403 5938 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176407 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176412 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176416 5938 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176420 5938 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:54:42.189680 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176425 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176429 5938 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176433 5938 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176437 5938 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176441 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176445 5938 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176449 5938 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176453 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176459 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176463 5938 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176467 5938 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176471 5938 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176475 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176480 5938 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176485 5938 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176489 5938 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176495 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176499 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176504 5938 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176508 5938 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:54:42.190318 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176512 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176516 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176520 5938 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176524 5938 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176529 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176533 5938 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176537 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176541 5938 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176545 5938 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176550 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176554 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176558 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176562 5938 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176567 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176571 5938 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176575 5938 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176579 5938 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176600 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176604 5938 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176609 5938 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:54:42.190872 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176615 5938 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176619 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176623 5938 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176627 5938 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176631 5938 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176635 5938 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176639 5938 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176645 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176650 5938 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176656 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176660 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176664 5938 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176668 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176672 5938 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176676 5938 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176683 5938 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176689 5938 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176693 5938 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176698 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:54:42.191351 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176702 5938 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176706 5938 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176710 5938 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176714 5938 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176718 5938 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176723 5938 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176727 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176732 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176736 5938 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176740 5938 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176744 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176751 5938 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176758 5938 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176764 5938 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176769 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176774 5938 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176778 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176782 5938 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176786 5938 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:54:42.191828 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176791 5938 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176795 5938 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176800 5938 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176806 5938 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176811 5938 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176815 5938 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176819 5938 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176823 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.176827 5938 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.180516 5938 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.180532 5938 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.180550 5938 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.180555 5938 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180627 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180634 5938 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180637 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180639 5938 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180642 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180645 5938 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:54:42.192284 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180648 5938 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180651 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180654 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180658 5938 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180662 5938 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180665 5938 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180667 5938 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180670 5938 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180673 5938 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180675 5938 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180678 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180681 5938 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180684 5938 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180687 5938 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180689 5938 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180692 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180694 5938 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180697 5938 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180699 5938 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:54:42.192874 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180702 5938 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180704 5938 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180707 5938 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180710 5938 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180713 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180716 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180718 5938 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180721 5938 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180723 5938 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180726 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180728 5938 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180731 5938 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180733 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180736 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180738 5938 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180740 5938 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180743 5938 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180745 5938 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180747 5938 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:54:42.193327 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180751 5938 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180755 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180758 5938 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180761 5938 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180763 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180766 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180768 5938 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180771 5938 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180773 5938 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180776 5938 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180779 5938 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180781 5938 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180784 5938 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180786 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180788 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180791 5938 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180793 5938 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180796 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180798 5938 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180801 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:54:42.193808 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180804 5938 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180806 5938 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180809 5938 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180811 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180814 5938 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180817 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180820 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180822 5938 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180825 5938 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180828 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180830 5938 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180832 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180835 5938 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180837 5938 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180839 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180842 5938 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180844 5938 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180847 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180850 5938 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:54:42.194281 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180852 5938 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180855 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180858 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180860 5938 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180863 5938 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180865 5938 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180868 5938 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180870 5938 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180873 5938 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180875 5938 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180877 5938 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180880 5938 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180883 5938 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180885 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180888 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180890 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180893 5938 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180895 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180898 5938 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:54:42.194757 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180901 5938 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:54:42.195234 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180903 5938 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:54:42.195234 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180906 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:54:42.195234 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.180908 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:54:42.195234 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.180935 5938 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:54:42.195234 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181063 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:54:42.195234 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181067 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:54:42.195234 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181070 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:54:42.195234 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181074 5938 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:54:42.195234 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181078 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:54:42.195234 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181081 5938 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:54:42.195234 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181084 5938 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:54:42.195234 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181087 5938 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:54:42.195234 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181090 5938 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:54:42.195234 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181093 5938 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:54:42.195234 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181095 5938 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181098 5938 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181100 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181103 5938 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181105 5938 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181108 5938 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181110 5938 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181112 5938 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181115 5938 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181117 5938 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181120 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181123 5938 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181125 5938 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181127 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181130 5938 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181132 5938 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181135 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181137 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181139 5938 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181142 5938 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:54:42.195731 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181144 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181147 5938 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181149 5938 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181151 5938 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181155 5938 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181158 5938 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181161 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181163 5938 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181166 5938 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181168 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181170 5938 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181174 5938 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181176 5938 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181179 5938 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181181 5938 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181184 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181186 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181189 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181191 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:54:42.196199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181193 5938 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181196 5938 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181198 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181201 5938 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181203 5938 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181207 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181209 5938 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181212 5938 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181214 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181217 5938 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181219 5938 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181221 5938 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181224 5938 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181226 5938 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181228 5938 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181231 5938 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181233 5938 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181236 5938 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181238 5938 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181240 5938 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:54:42.196666 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181243 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181245 5938 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181248 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181250 5938 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181253 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181255 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181258 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181260 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181262 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181265 5938 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181267 5938 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181270 5938 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181273 5938 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181275 5938 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181277 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181280 5938 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181282 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181284 5938 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181287 5938 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:54:42.197199 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181290 5938 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181292 5938 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181294 5938 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181297 5938 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181299 5938 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181301 5938 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181306 5938 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181308 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181311 5938 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181313 5938 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181315 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181318 5938 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181320 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181323 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181325 5938 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181328 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181330 5938 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:54:42.197687 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181333 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:54:42.198085 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.181356 5938 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:54:42.198085 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181392 5938 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:54:42.198085 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181396 5938 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:54:42.198085 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181398 5938 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:54:42.198085 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181401 5938 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:54:42.198085 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181403 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:54:42.198085 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181406 5938 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:54:42.198085 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181408 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:54:42.198085 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181411 5938 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:54:42.198085 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181413 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:54:42.198085 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181416 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:54:42.198085 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181418 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:54:42.198085 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181420 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:54:42.198085 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181423 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:54:42.198085 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181426 5938 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:54:42.198085 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181429 5938 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181431 5938 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181434 5938 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181436 5938 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181439 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181443 5938 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181445 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181448 5938 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181450 5938 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181453 5938 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181455 5938 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181457 5938 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181460 5938 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181465 5938 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181467 5938 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181470 5938 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181472 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181474 5938 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181477 5938 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181479 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:54:42.198469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181482 5938 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181484 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181487 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181489 5938 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181492 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181494 5938 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181496 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181499 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181501 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181504 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181508 5938 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181511 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181515 5938 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181518 5938 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181520 5938 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181523 5938 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181526 5938 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181530 5938 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181532 5938 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:54:42.199011 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181535 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181537 5938 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181540 5938 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181542 5938 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181544 5938 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181547 5938 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181551 5938 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181554 5938 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181556 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181559 5938 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181561 5938 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181564 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181566 5938 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181569 5938 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181571 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181573 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181576 5938 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181578 5938 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181581 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181600 5938 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:54:42.199469 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181604 5938 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181606 5938 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181610 5938 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181613 5938 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181616 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181620 5938 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181626 5938 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181629 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181632 5938 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181634 5938 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181638 5938 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181641 5938 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181643 5938 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181645 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181648 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181651 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181653 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181656 5938 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181660 5938 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:54:42.200063 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181662 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181665 5938 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181667 5938 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181670 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181674 5938 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181676 5938 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181679 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181681 5938 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181684 5938 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181686 5938 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181688 5938 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181691 5938 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181694 5938 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:54:42.181696 5938 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.181743 5938 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.182714 5938 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.182758 5938 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.182933 5938 server.go:1014] "Starting client certificate rotation" Oct 30 08:54:42.200552 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.183069 5938 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 08:54:42.201077 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.183109 5938 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 08:48:31 +0000 UTC" deadline="2028-07-27 18:04:39.333424876 +0000 UTC" Oct 30 08:54:42.201077 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.183144 5938 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15273h9m57.150282786s" Oct 30 08:54:42.201077 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.183342 5938 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 08:54:42.201077 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.184433 5938 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 08:54:42.201077 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.185938 5938 log.go:25] "Validated CRI v1 runtime API" Oct 30 08:54:42.201077 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.188340 5938 log.go:25] "Validated CRI v1 image API" Oct 30 08:54:42.201077 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.189095 5938 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 08:54:42.201077 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.192023 5938 fs.go:135] Filesystem UUIDs: map[2f8ac2d6-f89e-402b-b321-a5b8c8b6d6c1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fb73dd3f-f069-419b-b592-f501e3eab477:/dev/nvme0n1p3] Oct 30 08:54:42.201315 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.192038 5938 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/031cb85b6c9e79a498f26ef5470eacf148908a016f0fe2ca4c0806618779b30d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/031cb85b6c9e79a498f26ef5470eacf148908a016f0fe2ca4c0806618779b30d/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16e37ee9a32fff5b4adb761e178a3c321d8cb83004ac2911e5a304475f8947c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16e37ee9a32fff5b4adb761e178a3c321d8cb83004ac2911e5a304475f8947c2/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16ed60df4ee79e43d7ef21373cf8cef9830cec68ffb662f192083bbec7dff6fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16ed60df4ee79e43d7ef21373cf8cef9830cec68ffb662f192083bbec7dff6fd/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/231a4dc4aa0ceb519b5a4bb8c172528691b400defe5efdf373b307c25e222cdb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/231a4dc4aa0ceb519b5a4bb8c172528691b400defe5efdf373b307c25e222cdb/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/295f318365e4eb99495f4a14fb0db2dd7ff26288014604e8c758f50e8d46cce5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/295f318365e4eb99495f4a14fb0db2dd7ff26288014604e8c758f50e8d46cce5/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a3783a7cc9d0b76dc3550f1418f8bc2e1d98f5ff5acc882644ad9dcdce5e69c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a3783a7cc9d0b76dc3550f1418f8bc2e1d98f5ff5acc882644ad9dcdce5e69c/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2dd179b53efad516bbc24bd56ba672e4606abc1638c1eeed9363a97b5cde81a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2dd179b53efad516bbc24bd56ba672e4606abc1638c1eeed9363a97b5cde81a8/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/479a3215da80ed97eff8e2de3caf716c5bfae487c1b71dca238320902c5dae59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/479a3215da80ed97eff8e2de3caf716c5bfae487c1b71dca238320902c5dae59/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ca0a097a365338c29c4ce0db210c8da5e34c1079d3c0a8264abf33f3c055ca0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ca0a097a365338c29c4ce0db210c8da5e34c1079d3c0a8264abf33f3c055ca0/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d403b9188ea911ab69bfac4f21afc5f7177a9afcb5bb35b6d93677935e996cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d403b9188ea911ab69bfac4f21afc5f7177a9afcb5bb35b6d93677935e996cf/userdata/shm major:0 minor:493 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9cfd9aa07e797b37a7411dda20860cbee1995353ef4e8a79b3df01affd76cb19/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9cfd9aa07e797b37a7411dda20860cbee1995353ef4e8a79b3df01affd76cb19/userdata/shm major:0 minor:275 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1122495636b6dc03c9ece1042e2485fa558d1c4a26618100434e18a196d9587/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1122495636b6dc03c9ece1042e2485fa558d1c4a26618100434e18a196d9587/userdata/shm major:0 minor:279 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5ce646bfb584a3f1f9cbc2a1586240c601974f001775e87d087fcef54c40442/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5ce646bfb584a3f1f9cbc2a1586240c601974f001775e87d087fcef54c40442/userdata/shm major:0 minor:496 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5edfa0961397ed24bc8093e9253a61ceb6ef2f303041c738a5e27ba30f0487b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5edfa0961397ed24bc8093e9253a61ceb6ef2f303041c738a5e27ba30f0487b/userdata/shm major:0 minor:281 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afe2b09f39c62c80133b72645d1be31061be5cc1e171d66b4cb0067ce45d46b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afe2b09f39c62c80133b72645d1be31061be5cc1e171d66b4cb0067ce45d46b0/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c58a20b1277fdf78ca2e65a4053354b687b76f017e10245fbaf731cdea70fd5f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c58a20b1277fdf78ca2e65a4053354b687b76f017e10245fbaf731cdea70fd5f/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf6dd05fa5152d6207f48f8a0c192b5d84b9b8f376bd4300bc351fa6ced996db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf6dd05fa5152d6207f48f8a0c192b5d84b9b8f376bd4300bc351fa6ced996db/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cff2c8fb73b97f43e415a94fbbf0774aca7c04bde33f4c8e0337d8ee65ecaaf3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cff2c8fb73b97f43e415a94fbbf0774aca7c04bde33f4c8e0337d8ee65ecaaf3/userdata/shm major:0 minor:377 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1884abd5bd8c6e79a33ddd9ed0fc18f5be091a41614d0942abdebf7bc5b5553/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1884abd5bd8c6e79a33ddd9ed0fc18f5be091a41614d0942abdebf7bc5b5553/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de77cd9d9fb7cca1b8e6906c237d1a8e485707520566a42a75f7953b47f75707/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de77cd9d9fb7cca1b8e6906c237d1a8e485707520566a42a75f7953b47f75707/userdata/shm major:0 minor:495 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de8e1fa5a3d2bd96f1b430825b435f48cf9617c4ea7431323c387f9d1daddfe2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de8e1fa5a3d2bd96f1b430825b435f48cf9617c4ea7431323c387f9d1daddfe2/userdata/shm major:0 minor:439 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6ee89a073f2369082eb1f51a558980516571acf01d445de228953a6a7804334/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6ee89a073f2369082eb1f51a558980516571acf01d445de228953a6a7804334/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7e3415c41fdab90415e268f8425d6cb2f322f1efec0a924e2e5c8f878c6462f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7e3415c41fdab90415e268f8425d6cb2f322f1efec0a924e2e5c8f878c6462f/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa58aa796ef33d4bb7de2454907e70dc5379b2f40dbfd2930493687ad2f9fd65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa58aa796ef33d4bb7de2454907e70dc5379b2f40dbfd2930493687ad2f9fd65/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ecaa283-05a2-4f68-857e-dd14c857c804/volumes/kubernetes.io~projected/kube-api-access-l75cz:{mountpoint:/var/lib/kubelet/pods/1ecaa283-05a2-4f68-857e-dd14c857c804/volumes/kubernetes.io~projected/kube-api-access-l75cz major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ecaa283-05a2-4f68-857e-dd14c857c804/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/1ecaa283-05a2-4f68-857e-dd14c857c804/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ecaa283-05a2-4f68-857e-dd14c857c804/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/1ecaa283-05a2-4f68-857e-dd14c857c804/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25cb63c1-5df3-4be8-b407-b9bf2641cf95/volumes/kubernetes.io~projected/kube-api-access-szrjj:{mountpoint:/var/lib/kubelet/pods/25cb63c1-5df3-4be8-b407-b9bf2641cf95/volumes/kubernetes.io~projected/kube-api-access-szrjj major:0 minor:492 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b94c3af-6055-4f33-bd4b-57d4411d0b28/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2b94c3af-6055-4f33-bd4b-57d4411d0b28/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b94c3af-6055-4f33-bd4b-57d4411d0b28/volumes/kubernetes.io~projected/kube-api-access-m49gk:{mountpoint:/var/lib/kubelet/pods/2b94c3af-6055-4f33-bd4b-57d4411d0b28/volumes/kubernetes.io~projected/kube-api-access-m49gk major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b94c3af-6055-4f33-bd4b-57d4411d0b28/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2b94c3af-6055-4f33-bd4b-57d4411d0b28/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2eebeab0-d5a7-443d-9ed8-365761816c98/volumes/kubernetes.io~projected/kube-api-access-wgbtw:{mountpoint:/var/lib/kubelet/pods/2eebeab0-d5a7-443d-9ed8-365761816c98/volumes/kubernetes.io~projected/kube-api-access-wgbtw major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e6de072-eb99-48f0-ade6-f7fad4be241b/volumes/kubernetes.io~projected/kube-api-access-c94rp:{mountpoint:/var/lib/kubelet/pods/3e6de072-eb99-48f0-ade6-f7fad4be241b/volumes/kubernetes.io~projected/kube-api-access-c94rp major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e6de072-eb99-48f0-ade6-f7fad4be241b/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/3e6de072-eb99-48f0-ade6-f7fad4be241b/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e6de072-eb99-48f0-ade6-f7fad4be241b/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/3e6de072-eb99-48f0-ade6-f7fad4be241b/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e6de072-eb99-48f0-ade6-f7fad4be241b/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3e6de072-eb99-48f0-ade6-f7fad4be241b/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e6de072-eb99-48f0-ade6-f7fad4be241b/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/3e6de072-eb99-48f0-ade6-f7fad4be241b/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a763f73-62ad-43ed-babf-fb2ced67616a/volumes/kubernetes.io~projected/kube-api-access-hn5tj:{mountpoint:/var/lib/kubelet/pods/6a763f73-62ad-43ed-babf-fb2ced67616a/volumes/kubernetes.io~projected/kube-api-access-hn5tj major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a763f73-62ad-43ed-babf-fb2ced67616a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/6a763f73-62ad-43ed-babf-fb2ced67616a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a763f73-62ad-43ed-babf-fb2ced67616a/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/6a763f73-62ad-43ed-babf-fb2ced67616a/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334/volumes/kubernetes.io~projected/kube-api-access-fqch6:{mountpoint:/var/lib/kubelet/pods/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334/volumes/kubernetes.io~projected/kube-api-access-fqch6 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~projected/kube-api-access-9984j:{mountpoint:/var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~projected/kube-api-access-9984j major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c/volumes/kubernetes.io~projected/kube-api-access-vkdr4:{mountpoint:/var/lib/kubelet/pods/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c/volumes/kubernetes.io~projected/kube-api-access-vkdr4 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/866d9622-0c68-4e07-94ef-94cabbd34f86/volumes/kubernetes.io~projected/kube-api-access-bvg8d:{mountpoint:/var/lib/kubelet/pods/866d9622-0c68-4e07-94ef-94cabbd34f86/volumes/kubernetes.io~projected/kube-api-access-bvg8d major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/866d9622-0c68-4e07-94ef-94cabbd34f86/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/866d9622-0c68-4e07-94ef-94cabbd34f86/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/866d9622-0c68-4e07-94ef-94cabbd34f86/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/866d9622-0c68-4e07-94ef-94cabbd34f86/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f93427-45fe-46a3-8b9f-23d865534d41/volumes/kubernetes.io~projected/kube-api-access-kgsqg:{mountpoint:/var/lib/kubelet/pods/87f93427-45fe-46a3-8b9f-23d865534d41/volumes/kubernetes.io~projected/kube-api-access-kgsqg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volume-subpaths/web-config/config-reloader/2 major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volume-subpaths/web-config/prometheus/15 major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~empty-dir/config-out major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~projected/kube-api-access-tx5wx:{mountpoint:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~projected/kube-api-access-tx5wx major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~projected/tls-assets major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/config major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:433 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/web-config major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab3d086c-96af-4210-abf6-0cfbfe3afb20/volumes/kubernetes.io~projected/kube-api-access-sjskv:{mountpoint:/var/lib/kubelet/pods/ab3d086c-96af-4210-abf6-0cfbfe3afb20/volumes/kubernetes.io~projected/kube-api-access-sjskv major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab3d086c-96af-4210-abf6-0cfbfe3afb20/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/ab3d086c-96af-4210-abf6-0cfbfe3afb20/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volume-subpaths/web-config/alertmanager/9 major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volume-subpaths/web-config/config-reloader/6 major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~empty-dir/config-out major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~projected/kube-api-access-j75nb:{mountpoint:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~projected/kube-api-access-j75nb major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~projected/tls-assets major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/config-volume major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/web-config major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6/volumes/kubernetes.io~projected/kube-api-access-56x5w:{mountpoint:/var/lib/kubelet/pods/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6/volumes/kubernetes.io~projected/kube-api-access-56x5w major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c032d4ff-047f-43a9-a27e-41affb60615d/volumes/kubernetes.io~projected/kube-api-access-fx2kb:{mountpoint:/var/lib/kubelet/pods/c032d4ff-047f-43a9-a27e-41affb60615d/volumes/kubernetes.io~projected/kube-api-access-fx2kb major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7400965-54de-4420-a62a-37baf38d949d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c7400965-54de-4420-a62a-37baf38d949d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:491 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d20f4074-b1a5-4116-ac25-2d8d4235b7b7/volumes/kubernetes.io~projected/kube-api-access-psn67:{mountpoint:/var/lib/kubelet/pods/d20f4074-b1a5-4116-ac25-2d8d4235b7b7/volumes/kubernetes.io~projected/kube-api-access-psn67 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e23f66da-2587-4ce8-bb3a-a344e5dad910/volumes/kubernetes.io~projected/kube-api-access-frlr5:{mountpoint:/var/lib/kubelet/pods/e23f66da-2587-4ce8-bb3a-a344e5dad910/volumes/kubernetes.io~projected/kube-api-access-frlr5 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e23f66da-2587-4ce8-bb3a-a344e5dad910/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e23f66da-2587-4ce8-bb3a-a344e5dad910/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e23f66da-2587-4ce8-bb3a-a344e5dad910/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/e23f66da-2587-4ce8-bb3a-a344e5dad910/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f64a1d53-38a0-4133-accb-411e2dc6fc64/volumes/kubernetes.io~projected/kube-api-access-tgcgt:{mountpoint:/var/lib/kubelet/pods/f64a1d53-38a0-4133-accb-411e2dc6fc64/volumes/kubernetes.io~projected/kube-api-access-tgcgt major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f64a1d53-38a0-4133-accb-411e2dc6fc64/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/f64a1d53-38a0-4133-accb-411e2dc6fc64/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f76551a8-8b0b-459d-86f5-84cf822e83b8/volumes/kubernetes.io~projected/kube-api-access-2hj9q:{mountpoint:/var/lib/kubelet/pods/f76551a8-8b0b-459d-86f5-84cf822e83b8/volumes/kubernetes.io~projected/kube-api-access-2hj9q major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f76551a8-8b0b-459d-86f5-84cf822e83b8/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/f76551a8-8b0b-459d-86f5-84cf822e83b8/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f82405fd-bde5-4428-8b23-2be99042feef/volumes/kubernetes.io~projected/kube-api-access-bmq2k:{mountpoint:/var/lib/kubelet/pods/f82405fd-bde5-4428-8b23-2be99042feef/volumes/kubernetes.io~projected/kube-api-access-bmq2k major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f82405fd-bde5-4428-8b23-2be99042feef/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/f82405fd-bde5-4428-8b23-2be99042feef/volumes/kubernetes.io~secret/metrics-certs major:0 minor:487 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f507cb462bcd2e6d3d01b507df0b3025e4412d089cefdba68ef089eb156f2d8c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/fb1e826b0e4d779d79b547a07bb2f926add11ac766f60e27aaca1d9215ea69d8/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7daaec0abe7323b89e39c6680114fbea0db6423ca65114d07ded4b4c613d5531/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/ceeb3e49437a219269c0498966985aafafb038c22946c812cb61e80e64061e7c/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/46eea86101a94a0fe928fee79779bc32678614503e1021f01c8bfb86448d91ac/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/764ae70ab3614909d00616abeda0e0d5d302afdf19e4fa262f1a4b09490e33d7/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/4ed00bfb0d09b50ae95b70382b8eb2c459426b8cfc876bb824aa0a2fce3d0c6e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/10f83d483485e2aee8be9d1e8a95ac05508d2ef56a5e30f44c5d9b4aa1873b58/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/7d5e3f761c36287247f8ced8b95fe53a86398f3d6c1e6c21c5225be9bf9ebe70/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/656ebb96889b7c68de8e86ea3fee64e340272f176a36f2e7bd56f2cd20f1f12b/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/883587e8feb35bb5ba7b3ea7db6d679168d583afe3b957fcf1b1fc812860fab0/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/d5e3d9f50b1f387e75bfa966f2537eb2928349f3b3a042c1b57ca77faf4f27ff/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/ba8db71197ac62860d13c4b57a87d6918b95ae415cfd2353ee3d1a1ae8857f03/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/12884f674f4d4a91e63a9365257a3717d1592a7ca72e1ec16319c3230856eb7e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/3149a51fb87ef67b311270f7af01cab45d6f638020ea22849dc50277155ef77e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/222c9cbba072d7f8d660cfcb50a7bcdd8df1cba92a4fdce266f2d05a15493903/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/54c6e4e1e737165316c212a2dbf6e0f7883167fcd1fae9c5f87b726ccedebfbe/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/fbd578bd7562a5ccd09424ae9366b935f2ad565179ed941c2c2e14b1b17e01b8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/fe5e263f125263fabc50083967b227d4be72bb134be6f665dcab63ec67a00104/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a1896d6bf34f168414a6785ff63e9e95ac98a1f0801f69d54cf80a94f315eedd/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/156f780ec9593fd7b6b43e47a73f31c861f2ca32b6e5ea8a258e96e963fb3b2b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/c4f3c2684844b023cfe05181f793e7dd5aa4d79a8d22a46684c6c3a4203777db/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/4b50b166e9f524c88d3b1b2301f61897191f02819df0129491a69ca962bbfca9/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/877b2de402f6b3197a2b3beca241b73d2ffc8f42e3b4b9de849c370c77e23607/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/f87704dc317027a21046f161370dda7b76762aee8bdc40e4a730cdfade627f52/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/d1d35c0539ed68a8a2a61cfe118bcdbe3baed2fbbca9fe6b18c50e37994f471f/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/8224ea07de5e0262a73a4da41b633d6c066f035eceba4c4d6bc929678069c997/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/1a3abcf5f1f368002c3d194aed754eea687ea359f0ad0383dff105551e3404e6/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/e75c51a71b2bebc59ef5c75e3043b670bed5635670b82aa2566f9c7239b45187/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/de263fb748328f699925cc97c8b204259fb66fce36eddb891e79f38f5f5b13a5/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/d559051595727f87627782648c02f8f61b28d085d039ed1e3f7182746ded5e10/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/a0398500c352582a98ac13b232dc8fcb6165039006361bf1a26693a8014c862b/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/0edba0c0861ffd2b72b39148a077ea3afd72bba8a7a2a87ec9c09166f27d7b8b/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/455b8ecc46ce3602902d88411d0a71b4b115c1bb7dcfc4fc605ec65613099bae/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/20b2327d1aff3905e1c8d46a19f1fe539dfb3c2d84a5cb0c2a38ea64a76c03c5/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/9b7a0a29f1cd27df8f4ca6ee5fc3ff8b63198518b211be981638669132b0198c/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/18a9ed1151207606c98d9eac2f3dccbc0c4d77b289c7688c9500ff851ae6bfce/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/b59d78fa3414d19e7b8b8be4d9fc30acf73e754356b3ed830cc55852d7e7399f/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/6719af2121376c2f8042565c179f40ac8bb0933bbda83b136d0b4eb0114211e4/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/298954c03c27e4249c1f27607690575c0a793ca6d11fc9d61fbde657b516baab/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/f6c6340fe4d1d67ae2cfa99931a52afbc5356e0bb18dfa41ed795cfdafc34a5f/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/95bffa31df09a715131de06d47344949274a2853c24719617cadbf7e03712d8c/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/a949e59ff89702e3bfb17ad0b8c8855524fb551250b62bc6a514587645a836fd/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/296fbca1f1526d80a0609ab2c759028591d47e23cf32811a2167015fe8c8fdf4/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/2d567db8cb5b9c9ecf16e695c3874f0afc83a9a43ab169a6e278cd7dc1a09aec/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/c6a279592e4e8699b004bc85198b1ee05ad58c125feaacaf1b074c6e503cf528/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/93a68a5d530db01c54b03e5282004598e2d721d3e7a3382b61643251763a7d3f/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/403b61fe68eaf469e306e1c453ba21c6efb198598cc152220dfd61cc927443c6/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/60f2e21dc587d3217e85a497320d922b62f79708a8c34906849c78d58e0bce73/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/9b0e159c6713821071990f175afb32a077617334b7fd4e4ad7928ebf9d6ef066/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/ea37c00daa29b26eb789f8a7983415863d76b2f16867a8b8a51584b24ce89519/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/ce9e3444798f70a6fcde112ad7da31051fb558e637b86e9cc38e6b5750503bd5/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/4ae235c17f1642aacf198f43f96369f6f54b4ab55762a3691705bcacd1fd826f/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/21cc3cd441b041c3cbcd0aefea12ef372cd01b73e71c14c4b37f1bc41c546589/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/51f5c9ef2c07b0f4d8f407cc8a4c69c75bee38e33422a6cbceebc8adb3b812a8/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/0ae4c7d1e2772f74046fab4ce2e3b4574fca2a40bbfb1a5800aee7c4aa83ba7f/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/995d63425fbe7e259120f7cf5d8545e592bb6ca3d3f682cf6ffcd51331503fba/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/744e6b3a8c95d112ae47becdd76f86581649083d15d18564d21080aa0cfabe1b/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/9cfad6dfe3dd4df9627d98631c9d5f8295ab71818f0a8f22b5e21c06bb619698/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/e7a685b8b545718a526d8d7fc1388d7dfaf6c02236353f5b481fc43269b450a6/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/0339135180af355a48e1b70d1fb7c66a4a1126219407f72817f68cbbf90390bb/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-445:{mountpoint:/var/lib/containers/storage/overlay/dc13813acf6e21ab6ce080b455f7e57010710d99f0e1afa3fcd097a462aaea25/merged major:0 minor:445 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/35aad7d92b8ea25dc84a737e6e04ecdac5d8d39afcf4594a7a2873a0ab5c64d0/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/ed282107cebeb28c766cad0c4ccf0272a2a40021009618ff771ff53dcff57ce3/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-461:{mountpoint:/var/lib/containers/storage/overlay/a14314cb77f5bf985889a67757765ef74d187a2dd3504d92f8fd29d78c90168c/merged major:0 minor:461 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/c9c7a96a47b99a4ce49fd9732d47fed931b9f3d553bebf948712ce3bc0b64b7d/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-472:{mountpoint:/var/lib/containers/storage/overlay/8ca0625e34d12df48dac073f1142859cef40c0594220e089cdbcf53d6d595d27/merged major:0 minor:472 fsType:overlay blockSize:0} overlay_0-477:{mountpoint:/var/lib/containers/storage/overlay/ee2b3fe77e418eacf6230642242d2252ca78803ac1b15cff9a3fc1cdca6caf5c/merged major:0 minor:477 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/6d973d8e21e18c607a683358e13e942ef4dde930e9a8da6c2ca4286904a922cb/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-499:{mountpoint:/var/lib/containers/storage/overlay/b9ced3db84fed42759e2cbe0622855a78f6492cb16c17019d4b7fdb565798ed8/merged major:0 minor:499 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/69b86170eb486e62c989f20331e87bb409555c774d65f8b79e9b409ca63ace1d/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/08a25102b092e7d51920f3b22a372877379f10378bae7001d19fade5a3d37694/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-505:{mountpoint:/var/lib/containers/storage/overlay/fc4472bbf1df4dbe8f8f7af779eba3c6f0ca55d3cbce4321277b2f2cef609ce0/merged major:0 minor:505 fsType:overlay blockSize:0} overlay_0-507:{mountpoint:/var/lib/containers/storage/overlay/7c6898e0c489b0c4010cf89e9040d720006dcdbc9f863d7545f583fe42534443/merged major:0 minor:507 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/8b17d0e58d71b6034c2f6ffeded6eb55aa5c580f1c1c13134bb53b5fa002754b/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-513:{mountpoint:/var/lib/containers/storage/overlay/02cac8b0f70b6cebed753ec6ac3d16f2fec7bf05cf75d9c55ef3b2a36e2c42a0/merged major:0 minor:513 fsType:overlay blockSize:0} overlay_0-518:{mountpoint:/var/lib/containers/storage/overlay/d188a7d36cd05aaf749eb60d4df5765f532780a7377f8214f3c1632e593559a7/merged major:0 minor:518 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0613c727afbbf48ad0777ed00cdde83f0bc0960bcbe576110e38c314282d391e/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/9fdd98fea5c593fc015f9c8a955960ca14e55290a22571f4ccf3c6267e7eff8e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/37dc380f9d938a9b577f0c1d56c2f77814bafb08d85cd2ecd2cc7b15943c1d3c/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/bddbe916f57939af392ca1fc5913fa01e81a6ae3e83b9550ada71ce7963bd27a/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/036b7c2b4428442b978f700ac89911bd3fb086cf7a37e533d23d3243eb32431e/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/135ca82f21d1c741bb969dc8229a43ede58bfe5e64706d440a7c3723302cab8c/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/b2b609fdda12a3e383e4a78ad9d1e8978ecca06a6d66dac183c3b78055d9a285/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/6cf4ae8df37b9087e5311a640faf903e59be06a82d0825abb960cbadb64ce777/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/f00bea17f56f0488be48a869067ecea11e1954a3bbd922348ec6a1b498557052/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.199406 5938 manager.go:217] Machine: {Timestamp:2026-10-30 08:54:42.198962393 +0000 UTC m=+0.061668204 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091055 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26215c761113342d327dfe5c750350 SystemUUID:ec26215c-7611-1334-2d32-7dfe5c750350 BootID:8a3299f6-62fb-4bbc-9e80-3158f1cc740b Filesystems:[{Device:/var/lib/kubelet/pods/866d9622-0c68-4e07-94ef-94cabbd34f86/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:264 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:347 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:430 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:424 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de77cd9d9fb7cca1b8e6906c237d1a8e485707520566a42a75f7953b47f75707/userdata/shm DeviceMajor:0 DeviceMinor:495 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334/volumes/kubernetes.io~projected/kube-api-access-fqch6 DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2b94c3af-6055-4f33-bd4b-57d4411d0b28/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e6de072-eb99-48f0-ade6-f7fad4be241b/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:373 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:431 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:424 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-499 DeviceMajor:0 DeviceMinor:499 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16e37ee9a32fff5b4adb761e178a3c321d8cb83004ac2911e5a304475f8947c2/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6a763f73-62ad-43ed-babf-fb2ced67616a/volumes/kubernetes.io~projected/kube-api-access-hn5tj DeviceMajor:0 DeviceMinor:276 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~projected/kube-api-access-9984j DeviceMajor:0 DeviceMinor:349 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:303 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:302 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-477 DeviceMajor:0 DeviceMinor:477 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:305 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-507 DeviceMajor:0 DeviceMinor:507 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/031cb85b6c9e79a498f26ef5470eacf148908a016f0fe2ca4c0806618779b30d/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cff2c8fb73b97f43e415a94fbbf0774aca7c04bde33f4c8e0337d8ee65ecaaf3/userdata/shm DeviceMajor:0 DeviceMinor:377 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:424 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:346 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-461 DeviceMajor:0 DeviceMinor:461 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f64a1d53-38a0-4133-accb-411e2dc6fc64/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:343 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:345 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3e6de072-eb99-48f0-ade6-f7fad4be241b/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:374 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:306 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c/volumes/kubernetes.io~projected/kube-api-access-vkdr4 DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:436 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f82405fd-bde5-4428-8b23-2be99042feef/volumes/kubernetes.io~projected/kube-api-access-bmq2k DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f76551a8-8b0b-459d-86f5-84cf822e83b8/volumes/kubernetes.io~projected/kube-api-access-2hj9q DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afe2b09f39c62c80133b72645d1be31061be5cc1e171d66b4cb0067ce45d46b0/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e23f66da-2587-4ce8-bb3a-a344e5dad910/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~projected/kube-api-access-j75nb DeviceMajor:0 DeviceMinor:309 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:424 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~projected/kube-api-access-tx5wx DeviceMajor:0 DeviceMinor:438 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/295f318365e4eb99495f4a14fb0db2dd7ff26288014604e8c758f50e8d46cce5/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f76551a8-8b0b-459d-86f5-84cf822e83b8/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7400965-54de-4420-a62a-37baf38d949d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:491 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6a763f73-62ad-43ed-babf-fb2ced67616a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:274 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5ce646bfb584a3f1f9cbc2a1586240c601974f001775e87d087fcef54c40442/userdata/shm DeviceMajor:0 DeviceMinor:496 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-505 DeviceMajor:0 DeviceMinor:505 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-518 DeviceMajor:0 DeviceMinor:518 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9cfd9aa07e797b37a7411dda20860cbee1995353ef4e8a79b3df01affd76cb19/userdata/shm DeviceMajor:0 DeviceMinor:275 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:428 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1122495636b6dc03c9ece1042e2485fa558d1c4a26618100434e18a196d9587/userdata/shm DeviceMajor:0 DeviceMinor:279 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:303 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d20f4074-b1a5-4116-ac25-2d8d4235b7b7/volumes/kubernetes.io~projected/kube-api-access-psn67 DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:434 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:437 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2b94c3af-6055-4f33-bd4b-57d4411d0b28/volumes/kubernetes.io~projected/kube-api-access-m49gk DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6ee89a073f2369082eb1f51a558980516571acf01d445de228953a6a7804334/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:433 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2eebeab0-d5a7-443d-9ed8-365761816c98/volumes/kubernetes.io~projected/kube-api-access-wgbtw DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e23f66da-2587-4ce8-bb3a-a344e5dad910/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/479a3215da80ed97eff8e2de3caf716c5bfae487c1b71dca238320902c5dae59/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ca0a097a365338c29c4ce0db210c8da5e34c1079d3c0a8264abf33f3c055ca0/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e6de072-eb99-48f0-ade6-f7fad4be241b/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:369 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:297 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:307 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-513 DeviceMajor:0 DeviceMinor:513 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c58a20b1277fdf78ca2e65a4053354b687b76f017e10245fbaf731cdea70fd5f/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a3783a7cc9d0b76dc3550f1418f8bc2e1d98f5ff5acc882644ad9dcdce5e69c/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e6de072-eb99-48f0-ade6-f7fad4be241b/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:375 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25cb63c1-5df3-4be8-b407-b9bf2641cf95/volumes/kubernetes.io~projected/kube-api-access-szrjj DeviceMajor:0 DeviceMinor:492 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/866d9622-0c68-4e07-94ef-94cabbd34f86/volumes/kubernetes.io~projected/kube-api-access-bvg8d DeviceMajor:0 DeviceMinor:270 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:308 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3e6de072-eb99-48f0-ade6-f7fad4be241b/volumes/kubernetes.io~projected/kube-api-access-c94rp DeviceMajor:0 DeviceMinor:376 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:435 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-445 DeviceMajor:0 DeviceMinor:445 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-472 DeviceMajor:0 DeviceMinor:472 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ecaa283-05a2-4f68-857e-dd14c857c804/volumes/kubernetes.io~projected/kube-api-access-l75cz DeviceMajor:0 DeviceMinor:277 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c032d4ff-047f-43a9-a27e-41affb60615d/volumes/kubernetes.io~projected/kube-api-access-fx2kb DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa58aa796ef33d4bb7de2454907e70dc5379b2f40dbfd2930493687ad2f9fd65/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab3d086c-96af-4210-abf6-0cfbfe3afb20/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ab3d086c-96af-4210-abf6-0cfbfe3afb20/volumes/kubernetes.io~projected/kube-api-access-sjskv DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf6dd05fa5152d6207f48f8a0c192b5d84b9b8f376bd4300bc351fa6ced996db/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/866d9622-0c68-4e07-94ef-94cabbd34f86/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:344 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:429 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7e3415c41fdab90415e268f8425d6cb2f322f1efec0a924e2e5c8f878c6462f/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6/volumes/kubernetes.io~projected/kube-api-access-56x5w DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1884abd5bd8c6e79a33ddd9ed0fc18f5be091a41614d0942abdebf7bc5b5553/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f64a1d53-38a0-4133-accb-411e2dc6fc64/volumes/kubernetes.io~projected/kube-api-access-tgcgt DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ecaa283-05a2-4f68-857e-dd14c857c804/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:271 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:303 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:303 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/87f93427-45fe-46a3-8b9f-23d865534d41/volumes/kubernetes.io~projected/kube-api-access-kgsqg DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a763f73-62ad-43ed-babf-fb2ced67616a/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:272 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:304 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/231a4dc4aa0ceb519b5a4bb8c172528691b400defe5efdf373b307c25e222cdb/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5edfa0961397ed24bc8093e9253a61ceb6ef2f303041c738a5e27ba30f0487b/userdata/shm DeviceMajor:0 DeviceMinor:281 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7c5d7c66-ce15-4d20-bc37-4d832f308d9f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:348 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de8e1fa5a3d2bd96f1b430825b435f48cf9617c4ea7431323c387f9d1daddfe2/userdata/shm DeviceMajor:0 DeviceMinor:439 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f82405fd-bde5-4428-8b23-2be99042feef/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:487 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ecaa283-05a2-4f68-857e-dd14c857c804/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:273 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b94c3af-6055-4f33-bd4b-57d4411d0b28/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2dd179b53efad516bbc24bd56ba672e4606abc1638c1eeed9363a97b5cde81a8/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e23f66da-2587-4ce8-bb3a-a344e5dad910/volumes/kubernetes.io~projected/kube-api-access-frlr5 DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:301 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:307 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:432 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d403b9188ea911ab69bfac4f21afc5f7177a9afcb5bb35b6d93677935e996cf/userdata/shm DeviceMajor:0 DeviceMinor:493 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16ed60df4ee79e43d7ef21373cf8cef9830cec68ffb662f192083bbec7dff6fd/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:16e37ee9a32fff5 MacAddress:26:42:65:ea:52:a8 Speed:10000 Mtu:8901} {Name:479a3215da80ed9 MacAddress:be:7f:45:74:ce:18 Speed:10000 Mtu:8901} {Name:7ca0a097a365338 MacAddress:92:04:ae:7f:60:8c Speed:10000 Mtu:8901} {Name:7d403b9188ea911 MacAddress:b2:ec:0c:b9:26:cb Speed:10000 Mtu:8901} {Name:9cfd9aa07e797b3 MacAddress:b6:2a:49:2e:4c:3b Speed:10000 Mtu:8901} {Name:a1122495636b6dc MacAddress:f6:d9:fa:60:d5:78 Speed:10000 Mtu:8901} {Name:a5ce646bfb584a3 MacAddress:0a:df:d3:9f:23:d2 Speed:10000 Mtu:8901} {Name:afe2b09f39c62c8 MacAddress:f2:1c:f0:17:55:43 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e6:33:d6:4d Speed:0 Mtu:9001} {Name:br-int MacAddress:be:9c:8d:25:91:a1 Speed:0 Mtu:8901} {Name:cf6dd05fa5152d6 MacAddress:b2:47:0f:57:8b:1e Speed:10000 Mtu:8901} {Name:cff2c8fb73b97f4 MacAddress:e6:c5:04:02:48:8d Speed:10000 Mtu:8901} {Name:de77cd9d9fb7cca MacAddress:0e:54:1a:6c:29:fe Speed:10000 Mtu:8901} {Name:de8e1fa5a3d2bd9 MacAddress:c2:cb:b1:f2:41:5b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e6:33:d6:4d Speed:0 Mtu:9001} {Name:f6ee89a073f2369 MacAddress:92:b8:3a:2d:87:b7 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:2a:19:b5:b5:1a:df Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6a:cc:35:c6:bd:33 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.199867 5938 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.199908 5938 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.200200 5938 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.200225 5938 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-148.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.200356 5938 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.200362 5938 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.200374 5938 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.200386 5938 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.200454 5938 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.200639 5938 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.200744 5938 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.200778 5938 kubelet.go:493] "Attempting to sync node with API server" Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.200789 5938 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.200805 5938 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.200814 5938 kubelet.go:405] "Adding apiserver pod source" Oct 30 08:54:42.201640 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.200822 5938 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 08:54:42.202438 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.201669 5938 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 08:54:42.202438 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.201681 5938 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 08:54:42.202438 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.202160 5938 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 08:54:42.205935 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.205752 5938 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 08:54:42.205935 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.205771 5938 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 08:54:42.205935 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.205909 5938 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 08:54:42.206382 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.206363 5938 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 08:54:42.206459 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.206406 5938 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 08:54:42.206543 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.206532 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 08:54:42.206623 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.206552 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 08:54:42.206623 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.206563 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 08:54:42.206623 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.206573 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 08:54:42.206623 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.206581 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 08:54:42.206623 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.206615 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 08:54:42.206623 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.206625 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 08:54:42.206937 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.206633 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 08:54:42.206937 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.206645 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 08:54:42.206937 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.206654 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 08:54:42.206937 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.206667 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 08:54:42.206937 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.206682 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 08:54:42.206937 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.206715 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 08:54:42.206937 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.206725 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 08:54:42.210043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.210027 5938 server.go:1263] "Started kubelet" Oct 30 08:54:42.210138 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.210098 5938 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 08:54:42.210480 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.210140 5938 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 08:54:42.210480 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.210255 5938 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 08:54:42.210671 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.210658 5938 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 08:54:42.210531 ip-10-0-135-148 systemd[1]: Started Kubernetes Kubelet. Oct 30 08:54:42.211639 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.211621 5938 server.go:317] "Adding debug handlers to kubelet server" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.214031 5938 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.214053 5938 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.214162 5938 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 08:48:33 +0000 UTC" deadline="2028-03-29 14:14:07.311768902 +0000 UTC" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.214204 5938 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12389h19m25.097568112s" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.214262 5938 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.214268 5938 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:54:42.214274 5938 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.214378 5938 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.216403 5938 factory.go:55] Registering systemd factory Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.216421 5938 factory.go:223] Registration of the systemd container factory successfully Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.216740 5938 factory.go:153] Registering CRI-O factory Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.216752 5938 factory.go:223] Registration of the crio container factory successfully Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.216812 5938 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.216832 5938 factory.go:103] Registering Raw factory Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.216851 5938 manager.go:1199] Started watching for new ooms in manager Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217221 5938 manager.go:319] Starting recovery of all containers Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217669 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-db" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217702 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217718 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217732 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217745 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a763f73-62ad-43ed-babf-fb2ced67616a" volumeName="kubernetes.io/configmap/6a763f73-62ad-43ed-babf-fb2ced67616a-metrics-client-ca" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217761 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ac551a-4850-4c58-9fe8-6432e5d3301c" volumeName="kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217773 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ac551a-4850-4c58-9fe8-6432e5d3301c" volumeName="kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217785 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ac551a-4850-4c58-9fe8-6432e5d3301c" volumeName="kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217800 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e23f66da-2587-4ce8-bb3a-a344e5dad910" volumeName="kubernetes.io/projected/e23f66da-2587-4ce8-bb3a-a344e5dad910-kube-api-access-frlr5" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217822 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f64a1d53-38a0-4133-accb-411e2dc6fc64" volumeName="kubernetes.io/secret/f64a1d53-38a0-4133-accb-411e2dc6fc64-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217834 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f93427-45fe-46a3-8b9f-23d865534d41" volumeName="kubernetes.io/projected/87f93427-45fe-46a3-8b9f-23d865534d41-kube-api-access-kgsqg" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217849 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25cb63c1-5df3-4be8-b407-b9bf2641cf95" volumeName="kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217867 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b94c3af-6055-4f33-bd4b-57d4411d0b28" volumeName="kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovnkube-config" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217879 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e6de072-eb99-48f0-ade6-f7fad4be241b" volumeName="kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217894 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a763f73-62ad-43ed-babf-fb2ced67616a" volumeName="kubernetes.io/secret/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217906 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7400965-54de-4420-a62a-37baf38d949d" volumeName="kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217918 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e23f66da-2587-4ce8-bb3a-a344e5dad910" volumeName="kubernetes.io/secret/e23f66da-2587-4ce8-bb3a-a344e5dad910-prometheus-operator-tls" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217930 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ecaa283-05a2-4f68-857e-dd14c857c804" volumeName="kubernetes.io/secret/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217943 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c5d7c66-ce15-4d20-bc37-4d832f308d9f" volumeName="kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217955 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217966 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c032d4ff-047f-43a9-a27e-41affb60615d" volumeName="kubernetes.io/projected/c032d4ff-047f-43a9-a27e-41affb60615d-kube-api-access-fx2kb" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217979 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ecaa283-05a2-4f68-857e-dd14c857c804" volumeName="kubernetes.io/configmap/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.217992 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c5d7c66-ce15-4d20-bc37-4d832f308d9f" volumeName="kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218008 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218022 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ac551a-4850-4c58-9fe8-6432e5d3301c" volumeName="kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-main-db" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218043 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ac551a-4850-4c58-9fe8-6432e5d3301c" volumeName="kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-cluster-tls-config" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218058 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b94c3af-6055-4f33-bd4b-57d4411d0b28" volumeName="kubernetes.io/secret/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218073 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c5d7c66-ce15-4d20-bc37-4d832f308d9f" volumeName="kubernetes.io/projected/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-kube-api-access-9984j" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218095 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c5d7c66-ce15-4d20-bc37-4d832f308d9f" volumeName="kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-grpc-tls" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218111 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218126 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-grpc-tls" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218141 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218156 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6" volumeName="kubernetes.io/projected/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-kube-api-access-56x5w" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218171 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e23f66da-2587-4ce8-bb3a-a344e5dad910" volumeName="kubernetes.io/secret/e23f66da-2587-4ce8-bb3a-a344e5dad910-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218186 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b94c3af-6055-4f33-bd4b-57d4411d0b28" volumeName="kubernetes.io/projected/2b94c3af-6055-4f33-bd4b-57d4411d0b28-kube-api-access-m49gk" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218204 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e6de072-eb99-48f0-ade6-f7fad4be241b" volumeName="kubernetes.io/projected/3e6de072-eb99-48f0-ade6-f7fad4be241b-kube-api-access-c94rp" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218221 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70cee2c6-bc37-4ab5-9f5f-5ded85c0e334" volumeName="kubernetes.io/projected/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-kube-api-access-fqch6" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218236 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218252 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab3d086c-96af-4210-abf6-0cfbfe3afb20" volumeName="kubernetes.io/secret/ab3d086c-96af-4210-abf6-0cfbfe3afb20-cert" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218266 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d20f4074-b1a5-4116-ac25-2d8d4235b7b7" volumeName="kubernetes.io/projected/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-kube-api-access-psn67" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218282 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f64a1d53-38a0-4133-accb-411e2dc6fc64" volumeName="kubernetes.io/projected/f64a1d53-38a0-4133-accb-411e2dc6fc64-kube-api-access-tgcgt" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218298 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b94c3af-6055-4f33-bd4b-57d4411d0b28" volumeName="kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovnkube-script-lib" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218313 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eebeab0-d5a7-443d-9ed8-365761816c98" volumeName="kubernetes.io/projected/2eebeab0-d5a7-443d-9ed8-365761816c98-kube-api-access-wgbtw" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218331 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="866d9622-0c68-4e07-94ef-94cabbd34f86" volumeName="kubernetes.io/projected/866d9622-0c68-4e07-94ef-94cabbd34f86-kube-api-access-bvg8d" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218347 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218364 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ac551a-4850-4c58-9fe8-6432e5d3301c" volumeName="kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-metrics-client-ca" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218379 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ac551a-4850-4c58-9fe8-6432e5d3301c" volumeName="kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-tls-assets" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218395 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d20f4074-b1a5-4116-ac25-2d8d4235b7b7" volumeName="kubernetes.io/configmap/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-cni-binary-copy" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218409 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f64a1d53-38a0-4133-accb-411e2dc6fc64" volumeName="kubernetes.io/configmap/f64a1d53-38a0-4133-accb-411e2dc6fc64-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218421 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ecaa283-05a2-4f68-857e-dd14c857c804" volumeName="kubernetes.io/empty-dir/1ecaa283-05a2-4f68-857e-dd14c857c804-volume-directive-shadow" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218431 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e6de072-eb99-48f0-ade6-f7fad4be241b" volumeName="kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-telemeter-client-tls" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218441 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a763f73-62ad-43ed-babf-fb2ced67616a" volumeName="kubernetes.io/configmap/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218448 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c5d7c66-ce15-4d20-bc37-4d832f308d9f" volumeName="kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218455 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c032d4ff-047f-43a9-a27e-41affb60615d" volumeName="kubernetes.io/configmap/c032d4ff-047f-43a9-a27e-41affb60615d-serviceca" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218462 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e6de072-eb99-48f0-ade6-f7fad4be241b" volumeName="kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218468 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e6de072-eb99-48f0-ade6-f7fad4be241b" volumeName="kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-federate-client-tls" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218480 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a763f73-62ad-43ed-babf-fb2ced67616a" volumeName="kubernetes.io/empty-dir/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-textfile" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218486 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dee6e4b-1bc3-4718-a1a6-d268dd6b405c" volumeName="kubernetes.io/projected/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-kube-api-access-vkdr4" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218493 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218501 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ac551a-4850-4c58-9fe8-6432e5d3301c" volumeName="kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-kube-api-access-j75nb" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218509 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e23f66da-2587-4ce8-bb3a-a344e5dad910" volumeName="kubernetes.io/configmap/e23f66da-2587-4ce8-bb3a-a344e5dad910-metrics-client-ca" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218515 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f64a1d53-38a0-4133-accb-411e2dc6fc64" volumeName="kubernetes.io/empty-dir/f64a1d53-38a0-4133-accb-411e2dc6fc64-data-volume" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218522 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218528 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e6de072-eb99-48f0-ade6-f7fad4be241b" volumeName="kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-secret-telemeter-client" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218604 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eebeab0-d5a7-443d-9ed8-365761816c98" volumeName="kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218616 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e6de072-eb99-48f0-ade6-f7fad4be241b" volumeName="kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-metrics-client-ca" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218627 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c5d7c66-ce15-4d20-bc37-4d832f308d9f" volumeName="kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218636 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-kube-api-access-tx5wx" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218646 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ac551a-4850-4c58-9fe8-6432e5d3301c" volumeName="kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-out" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218655 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f82405fd-bde5-4428-8b23-2be99042feef" volumeName="kubernetes.io/projected/f82405fd-bde5-4428-8b23-2be99042feef-kube-api-access-bmq2k" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218665 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a763f73-62ad-43ed-babf-fb2ced67616a" volumeName="kubernetes.io/projected/6a763f73-62ad-43ed-babf-fb2ced67616a-kube-api-access-hn5tj" seLinuxMountContext="" Oct 30 08:54:42.218619 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218677 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a763f73-62ad-43ed-babf-fb2ced67616a" volumeName="kubernetes.io/secret/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-tls" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218687 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c5d7c66-ce15-4d20-bc37-4d832f308d9f" volumeName="kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218697 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="866d9622-0c68-4e07-94ef-94cabbd34f86" volumeName="kubernetes.io/secret/866d9622-0c68-4e07-94ef-94cabbd34f86-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218706 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ac551a-4850-4c58-9fe8-6432e5d3301c" volumeName="kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218716 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ac551a-4850-4c58-9fe8-6432e5d3301c" volumeName="kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-volume" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218726 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f76551a8-8b0b-459d-86f5-84cf822e83b8" volumeName="kubernetes.io/empty-dir/f76551a8-8b0b-459d-86f5-84cf822e83b8-tmp-dir" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218781 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f82405fd-bde5-4428-8b23-2be99042feef" volumeName="kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218794 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ecaa283-05a2-4f68-857e-dd14c857c804" volumeName="kubernetes.io/projected/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-api-access-l75cz" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218807 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b94c3af-6055-4f33-bd4b-57d4411d0b28" volumeName="kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-env-overrides" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218818 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dee6e4b-1bc3-4718-a1a6-d268dd6b405c" volumeName="kubernetes.io/empty-dir/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-tuned" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218829 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-tls-assets" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218838 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f76551a8-8b0b-459d-86f5-84cf822e83b8" volumeName="kubernetes.io/configmap/f76551a8-8b0b-459d-86f5-84cf822e83b8-config-volume" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218850 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ecaa283-05a2-4f68-857e-dd14c857c804" volumeName="kubernetes.io/secret/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218861 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eebeab0-d5a7-443d-9ed8-365761816c98" volumeName="kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218872 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c5d7c66-ce15-4d20-bc37-4d832f308d9f" volumeName="kubernetes.io/configmap/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-metrics-client-ca" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218882 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="866d9622-0c68-4e07-94ef-94cabbd34f86" volumeName="kubernetes.io/configmap/866d9622-0c68-4e07-94ef-94cabbd34f86-metrics-client-ca" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218893 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218903 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6" volumeName="kubernetes.io/empty-dir/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-tmp-dir" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218914 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f76551a8-8b0b-459d-86f5-84cf822e83b8" volumeName="kubernetes.io/secret/f76551a8-8b0b-459d-86f5-84cf822e83b8-metrics-tls" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218924 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e6de072-eb99-48f0-ade6-f7fad4be241b" volumeName="kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218935 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="866d9622-0c68-4e07-94ef-94cabbd34f86" volumeName="kubernetes.io/secret/866d9622-0c68-4e07-94ef-94cabbd34f86-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218947 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b" volumeName="kubernetes.io/configmap/8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b-konnectivity-ca" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218957 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-web-config" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218967 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218979 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab3d086c-96af-4210-abf6-0cfbfe3afb20" volumeName="kubernetes.io/projected/ab3d086c-96af-4210-abf6-0cfbfe3afb20-kube-api-access-sjskv" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218989 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ac551a-4850-4c58-9fe8-6432e5d3301c" volumeName="kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.218999 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f76551a8-8b0b-459d-86f5-84cf822e83b8" volumeName="kubernetes.io/projected/f76551a8-8b0b-459d-86f5-84cf822e83b8-kube-api-access-2hj9q" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.219010 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dee6e4b-1bc3-4718-a1a6-d268dd6b405c" volumeName="kubernetes.io/empty-dir/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-tmp" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.219021 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f93427-45fe-46a3-8b9f-23d865534d41" volumeName="kubernetes.io/configmap/87f93427-45fe-46a3-8b9f-23d865534d41-iptables-alerter-script" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.219031 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b" volumeName="kubernetes.io/secret/8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b-agent-certs" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.219043 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config-out" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.219062 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ac551a-4850-4c58-9fe8-6432e5d3301c" volumeName="kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-web-config" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.219073 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d20f4074-b1a5-4116-ac25-2d8d4235b7b7" volumeName="kubernetes.io/configmap/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-daemon-config" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.219085 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eebeab0-d5a7-443d-9ed8-365761816c98" volumeName="kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-cni-binary-copy" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.219098 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ecaa283-05a2-4f68-857e-dd14c857c804" volumeName="kubernetes.io/configmap/1ecaa283-05a2-4f68-857e-dd14c857c804-metrics-client-ca" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.219109 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c5d7c66-ce15-4d20-bc37-4d832f308d9f" volumeName="kubernetes.io/empty-dir/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-tmp" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.219122 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f15807d-cf9f-435a-84e6-febce2ef9ce4" volumeName="kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.219133 5938 reconstruct.go:97] "Volume reconstruction finished" Oct 30 08:54:42.222603 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.219140 5938 reconciler.go:29] "Reconciler: start to sync state" Oct 30 08:54:42.247403 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.247352 5938 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 08:54:42.248873 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.248853 5938 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 08:54:42.248873 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.248874 5938 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 08:54:42.249040 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.248895 5938 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 08:54:42.249040 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:54:42.248935 5938 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 08:54:42.258458 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.258421 5938 generic.go:356] "Generic (PLEG): container finished" podID="2eebeab0-d5a7-443d-9ed8-365761816c98" containerID="46a21a8ba482075246800ea45b1781f7ec06b7aecc30afc9e65b7368af39455b" exitCode=0 Oct 30 08:54:42.258458 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.258444 5938 generic.go:356] "Generic (PLEG): container finished" podID="2eebeab0-d5a7-443d-9ed8-365761816c98" containerID="c43d32e9fc013d3c2ffa6ba5fa6a3a5826f08692ebeac5b6169415b255edf233" exitCode=0 Oct 30 08:54:42.258458 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.258453 5938 generic.go:356] "Generic (PLEG): container finished" podID="2eebeab0-d5a7-443d-9ed8-365761816c98" containerID="b7c0a73e793951aac6b9aa847adc18c5d866e14950a5a85eff4576666119faff" exitCode=0 Oct 30 08:54:42.258458 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.258459 5938 generic.go:356] "Generic (PLEG): container finished" podID="2eebeab0-d5a7-443d-9ed8-365761816c98" containerID="d1ac84a97e6eb5b19831f97a86e4c4ea77354a78d65c0316689661ca8df92290" exitCode=0 Oct 30 08:54:42.258722 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.258468 5938 generic.go:356] "Generic (PLEG): container finished" podID="2eebeab0-d5a7-443d-9ed8-365761816c98" containerID="dcb47eae769f35e34138619f2af8ef61ab1f7ecec947d2b3b8a71aadb6d8fadf" exitCode=0 Oct 30 08:54:42.258722 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.258477 5938 generic.go:356] "Generic (PLEG): container finished" podID="2eebeab0-d5a7-443d-9ed8-365761816c98" containerID="944e4b896f61b55f39c727d0381bbeff779cba980d5f9a3cf572f1bdecb88822" exitCode=0 Oct 30 08:54:42.266697 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.266675 5938 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:54:42.267768 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.267735 5938 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:54:42.267861 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.267768 5938 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 08:54:42.272508 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.272487 5938 generic.go:356] "Generic (PLEG): container finished" podID="6a763f73-62ad-43ed-babf-fb2ced67616a" containerID="befc92f6f4d381e6b9be57d97653b7eaed1d0d894d8425b86494a64f33c96cd2" exitCode=0 Oct 30 08:54:42.280814 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.280795 5938 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:54:42.283788 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.283768 5938 generic.go:356] "Generic (PLEG): container finished" podID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerID="c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86" exitCode=0 Oct 30 08:54:42.292565 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.292543 5938 generic.go:356] "Generic (PLEG): container finished" podID="7f68f752438bee3aab3d894447acc3ea" containerID="b09aca77ac5af5ef49da1035110eea8ae02c22e96c7e08f7cf8ef11d21950ddc" exitCode=0 Oct 30 08:54:42.293704 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.293684 5938 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:54:42.310328 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.310305 5938 generic.go:356] "Generic (PLEG): container finished" podID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerID="428a41797386d649baf498c2ec11969043e66d787e4e260eb3d334cd7fc51391" exitCode=0 Oct 30 08:54:42.338966 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.338947 5938 manager.go:324] Recovery completed Oct 30 08:54:42.349026 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:54:42.348979 5938 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 08:54:42.366399 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.366377 5938 cpu_manager.go:226] "Starting" policy="none" Oct 30 08:54:42.366399 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.366391 5938 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 08:54:42.366536 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.366409 5938 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 08:54:42.366536 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.366520 5938 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 08:54:42.366645 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.366530 5938 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 08:54:42.366645 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.366545 5938 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 08:54:42.366645 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.366551 5938 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 08:54:42.366645 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.366555 5938 policy_none.go:50] "Start" Oct 30 08:54:42.366645 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.366563 5938 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 08:54:42.366645 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.366571 5938 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 08:54:42.366814 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.366720 5938 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 08:54:42.366814 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.366727 5938 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 08:54:42.366814 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.366733 5938 policy_none.go:44] "Start" Oct 30 08:54:42.371480 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.371448 5938 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 08:54:42.374724 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.374707 5938 manager.go:341] "Starting Device Plugin manager" Oct 30 08:54:42.375311 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:54:42.374731 5938 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 08:54:42.375311 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.375312 5938 server.go:87] "Starting device plugin registration server" Oct 30 08:54:42.375569 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.375556 5938 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 08:54:42.375664 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.375568 5938 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 08:54:42.375664 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.375645 5938 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 08:54:42.375783 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.375757 5938 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 08:54:42.375842 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.375800 5938 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 08:54:42.375842 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.375807 5938 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 08:54:42.375932 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.375867 5938 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T08:54:42.375782329Z","UUID":"26def66e-2319-472a-a6b7-454f72cfc211","Handler":null,"Name":"","Endpoint":""} Oct 30 08:54:42.377076 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.377059 5938 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 08:54:42.377177 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.377086 5938 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 08:54:42.377778 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:54:42.377758 5938 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 08:54:42.476180 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.476138 5938 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:54:42.477788 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.477774 5938 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:54:42.477855 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.477801 5938 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:54:42.477855 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.477811 5938 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:54:42.477855 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.477854 5938 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-148.ec2.internal" Oct 30 08:54:42.513702 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.513684 5938 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-148.ec2.internal" Oct 30 08:54:42.513749 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.513738 5938 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-148.ec2.internal" Oct 30 08:54:42.549672 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.549634 5938 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal"] Oct 30 08:54:42.550230 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.550204 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" event={"ID":"7f68f752438bee3aab3d894447acc3ea","Type":"ContainerStarted","Data":"56bd1ddca0149baca35b31eca5c5c77403b5084f722b9a38bd08930da5f25cbf"} Oct 30 08:54:42.550270 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.550237 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" event={"ID":"7f68f752438bee3aab3d894447acc3ea","Type":"ContainerDied","Data":"b09aca77ac5af5ef49da1035110eea8ae02c22e96c7e08f7cf8ef11d21950ddc"} Oct 30 08:54:42.550270 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.550251 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" event={"ID":"7f68f752438bee3aab3d894447acc3ea","Type":"ContainerStarted","Data":"f7e3415c41fdab90415e268f8425d6cb2f322f1efec0a924e2e5c8f878c6462f"} Oct 30 08:54:42.550270 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.550263 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" event={"ID":"9307784112acd4d13d34c64b57f18e71","Type":"ContainerStarted","Data":"b15d12f37e30e8e178ac969f728a57b6fd33b3661956e5a73f625546dd43ed35"} Oct 30 08:54:42.550392 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.550274 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" event={"ID":"9307784112acd4d13d34c64b57f18e71","Type":"ContainerStarted","Data":"16ed60df4ee79e43d7ef21373cf8cef9830cec68ffb662f192083bbec7dff6fd"} Oct 30 08:54:42.550392 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.550219 5938 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" Oct 30 08:54:42.550448 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.550434 5938 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:54:42.575736 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.575721 5938 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:54:42.575807 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:54:42.575753 5938 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-148.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" Oct 30 08:54:42.576108 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.576095 5938 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:54:42.576151 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:54:42.576131 5938 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:54:42.669458 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.669437 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9307784112acd4d13d34c64b57f18e71-config\") pod \"kube-apiserver-proxy-ip-10-0-135-148.ec2.internal\" (UID: \"9307784112acd4d13d34c64b57f18e71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" Oct 30 08:54:42.669537 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.669460 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f68f752438bee3aab3d894447acc3ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"7f68f752438bee3aab3d894447acc3ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:54:42.669537 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.669479 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f68f752438bee3aab3d894447acc3ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"7f68f752438bee3aab3d894447acc3ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:54:42.770035 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.770000 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f68f752438bee3aab3d894447acc3ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"7f68f752438bee3aab3d894447acc3ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:54:42.770035 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.770042 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f68f752438bee3aab3d894447acc3ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"7f68f752438bee3aab3d894447acc3ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:54:42.770169 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.770061 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9307784112acd4d13d34c64b57f18e71-config\") pod \"kube-apiserver-proxy-ip-10-0-135-148.ec2.internal\" (UID: \"9307784112acd4d13d34c64b57f18e71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" Oct 30 08:54:42.770169 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.770060 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f68f752438bee3aab3d894447acc3ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"7f68f752438bee3aab3d894447acc3ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:54:42.770169 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.770096 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9307784112acd4d13d34c64b57f18e71-config\") pod \"kube-apiserver-proxy-ip-10-0-135-148.ec2.internal\" (UID: \"9307784112acd4d13d34c64b57f18e71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" Oct 30 08:54:42.770169 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:42.770105 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f68f752438bee3aab3d894447acc3ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"7f68f752438bee3aab3d894447acc3ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:54:43.201285 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.201262 5938 apiserver.go:52] "Watching apiserver" Oct 30 08:54:43.259666 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.259639 5938 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 08:54:43.260393 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.260370 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-qgqjn","openshift-dns/node-resolver-q8z8s","openshift-multus/network-metrics-daemon-ktt7j","kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p","openshift-monitoring/alertmanager-main-0","openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx","openshift-ovn-kubernetes/ovnkube-node-mf5b5","kube-system/global-pull-secret-syncer-m5sz6","openshift-monitoring/prometheus-k8s-0","openshift-multus/multus-additional-cni-plugins-hwg67","openshift-image-registry/node-ca-mnq7z","openshift-ingress-canary/ingress-canary-kwlz8","openshift-cluster-node-tuning-operator/tuned-nt944","openshift-insights/insights-runtime-extractor-vbz6m","openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2","openshift-monitoring/prometheus-operator-9cd89b74c-z42cw","openshift-monitoring/node-exporter-8kgvx","kube-system/konnectivity-agent-hsqk7","openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg","openshift-monitoring/telemeter-client-65c85d64c4-d8zwz","openshift-network-diagnostics/network-check-target-tf9r8","openshift-network-operator/iptables-alerter-kft2v","openshift-dns/dns-default-fxq45"] Oct 30 08:54:43.279072 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.279044 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 08:54:43.279180 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.279146 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 08:54:43.279354 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.279343 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 08:54:43.279707 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.279692 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-6v9sh\"" Oct 30 08:54:43.279707 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.279695 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pw6p6\"" Oct 30 08:54:43.280445 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.280430 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 08:54:43.280531 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.280517 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kgscj\"" Oct 30 08:54:43.280806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.280784 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 08:54:43.280864 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.280851 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 08:54:43.280933 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.280922 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 08:54:43.280973 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.280962 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 08:54:43.281021 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.280992 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r2tvf\"" Oct 30 08:54:43.281059 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.281024 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 08:54:43.283050 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.283036 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qhn6k\"" Oct 30 08:54:43.302256 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.302237 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 08:54:43.316785 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.316767 5938 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 08:54:43.321004 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.320978 5938 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:54:43.321522 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.321505 5938 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" Oct 30 08:54:43.347184 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.347165 5938 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:54:43.347278 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:54:43.347210 5938 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" Oct 30 08:54:43.347382 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.347366 5938 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:54:43.347434 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:54:43.347400 5938 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-148.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" Oct 30 08:54:43.372464 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372443 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.372563 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372468 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-web-config\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.372563 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372485 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.372563 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372501 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.372563 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372526 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-etc-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.372563 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372554 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-run-ovn-kubernetes\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.372800 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372580 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgcgt\" (UniqueName: \"kubernetes.io/projected/f64a1d53-38a0-4133-accb-411e2dc6fc64-kube-api-access-tgcgt\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:43.372800 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372658 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.372800 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372689 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-socket-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.372800 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372737 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ecaa283-05a2-4f68-857e-dd14c857c804-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:43.372800 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372790 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.372967 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372833 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config-out\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.372967 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372870 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-cni-netd\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.372967 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372926 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config-out\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.372967 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372951 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovnkube-config\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.372967 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372968 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-sys\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.373137 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.372983 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-lib-modules\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.373137 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373010 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-sys-fs\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.373137 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373034 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.373137 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373102 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.373137 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373121 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.373296 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373136 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b-agent-certs\") pod \"konnectivity-agent-hsqk7\" (UID: \"8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b\") " pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:54:43.373296 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373155 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-netns\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.373296 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373170 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-kubelet\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.373296 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373187 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-telemeter-client-tls\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:43.373296 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373212 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-tmp\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.373296 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373289 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-device-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.373511 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373311 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-tmp\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.373511 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373353 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-cni-binary-copy\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.373511 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373360 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b-agent-certs\") pod \"konnectivity-agent-hsqk7\" (UID: \"8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b\") " pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:54:43.373511 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373392 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9984j\" (UniqueName: \"kubernetes.io/projected/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-kube-api-access-9984j\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.373511 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373435 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-systemd\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.373748 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373526 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-ovn\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.373748 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373563 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.373748 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373626 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.373748 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373665 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-cni-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.373748 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373690 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-cni-bin\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.373748 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373726 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovnkube-script-lib\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.373960 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373768 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/866d9622-0c68-4e07-94ef-94cabbd34f86-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:43.373960 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373803 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-cnibin\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.373960 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373828 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.373960 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373852 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-socket-dir-parent\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.373960 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373877 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a763f73-62ad-43ed-babf-fb2ced67616a-metrics-client-ca\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.373960 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373933 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.374213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.373972 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjskv\" (UniqueName: \"kubernetes.io/projected/ab3d086c-96af-4210-abf6-0cfbfe3afb20-kube-api-access-sjskv\") pod \"ingress-canary-kwlz8\" (UID: \"ab3d086c-96af-4210-abf6-0cfbfe3afb20\") " pod="openshift-ingress-canary/ingress-canary-kwlz8" Oct 30 08:54:43.374213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374021 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmq2k\" (UniqueName: \"kubernetes.io/projected/f82405fd-bde5-4428-8b23-2be99042feef-kube-api-access-bmq2k\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:43.374213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374108 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/87f93427-45fe-46a3-8b9f-23d865534d41-iptables-alerter-script\") pod \"iptables-alerter-kft2v\" (UID: \"87f93427-45fe-46a3-8b9f-23d865534d41\") " pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:54:43.374213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374135 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.374213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374172 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-volume\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.374213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374198 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-tls\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.374471 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374218 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.374471 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374236 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.374471 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374258 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f64a1d53-38a0-4133-accb-411e2dc6fc64-data-volume\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:43.374471 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374303 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e23f66da-2587-4ce8-bb3a-a344e5dad910-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:43.374471 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374320 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1ecaa283-05a2-4f68-857e-dd14c857c804-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:43.374471 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374334 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f64a1d53-38a0-4133-accb-411e2dc6fc64-data-volume\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:43.374471 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374345 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-k8s-cni-cncf-io\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.374471 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374376 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7400965-54de-4420-a62a-37baf38d949d-kubelet-config\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:43.374471 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374401 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkdr4\" (UniqueName: \"kubernetes.io/projected/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-kube-api-access-vkdr4\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.374471 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374406 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1ecaa283-05a2-4f68-857e-dd14c857c804-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:43.374471 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374428 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:43.374471 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374467 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374490 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-root\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374507 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f76551a8-8b0b-459d-86f5-84cf822e83b8-config-volume\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374525 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hj9q\" (UniqueName: \"kubernetes.io/projected/f76551a8-8b0b-459d-86f5-84cf822e83b8-kube-api-access-2hj9q\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374542 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab3d086c-96af-4210-abf6-0cfbfe3afb20-cert\") pod \"ingress-canary-kwlz8\" (UID: \"ab3d086c-96af-4210-abf6-0cfbfe3afb20\") " pod="openshift-ingress-canary/ingress-canary-kwlz8" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374578 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frlr5\" (UniqueName: \"kubernetes.io/projected/e23f66da-2587-4ce8-bb3a-a344e5dad910-kube-api-access-frlr5\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374628 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-systemd-units\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374659 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374700 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/866d9622-0c68-4e07-94ef-94cabbd34f86-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374716 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-tuned\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374753 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-tmp\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374787 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-cnibin\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374803 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-tuned\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374818 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-federate-client-tls\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374848 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87f93427-45fe-46a3-8b9f-23d865534d41-host-slash\") pod \"iptables-alerter-kft2v\" (UID: \"87f93427-45fe-46a3-8b9f-23d865534d41\") " pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374876 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e23f66da-2587-4ce8-bb3a-a344e5dad910-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374885 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-tmp\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374906 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374935 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx5wx\" (UniqueName: \"kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-kube-api-access-tx5wx\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374961 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-etc-kubernetes\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.374979 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psn67\" (UniqueName: \"kubernetes.io/projected/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-kube-api-access-psn67\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.375013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375004 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-tmp-dir\") pod \"node-resolver-q8z8s\" (UID: \"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6\") " pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375038 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-system-cni-dir\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375071 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbtw\" (UniqueName: \"kubernetes.io/projected/2eebeab0-d5a7-443d-9ed8-365761816c98-kube-api-access-wgbtw\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375072 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-tmp-dir\") pod \"node-resolver-q8z8s\" (UID: \"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6\") " pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375105 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-hostroot\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375125 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-run-netns\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375142 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-daemon-config\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375225 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvg8d\" (UniqueName: \"kubernetes.io/projected/866d9622-0c68-4e07-94ef-94cabbd34f86-kube-api-access-bvg8d\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375258 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375285 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375314 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-conf-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375399 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-node-log\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375429 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-serving-certs-ca-bundle\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375456 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375512 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgsqg\" (UniqueName: \"kubernetes.io/projected/87f93427-45fe-46a3-8b9f-23d865534d41-kube-api-access-kgsqg\") pod \"iptables-alerter-kft2v\" (UID: \"87f93427-45fe-46a3-8b9f-23d865534d41\") " pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375539 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-grpc-tls\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375555 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375636 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-var-lib-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375667 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-log-socket\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375713 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375784 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375830 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375856 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f64a1d53-38a0-4133-accb-411e2dc6fc64-crio-socket\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375874 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-textfile\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375895 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375911 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-metrics-client-ca\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375927 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.375929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375945 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f64a1d53-38a0-4133-accb-411e2dc6fc64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375969 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-textfile\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.375993 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-metrics-client-ca\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376012 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrjj\" (UniqueName: \"kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj\") pod \"network-check-target-tf9r8\" (UID: \"25cb63c1-5df3-4be8-b407-b9bf2641cf95\") " pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376035 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e23f66da-2587-4ce8-bb3a-a344e5dad910-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376060 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-slash\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376083 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7400965-54de-4420-a62a-37baf38d949d-dbus\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376111 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-secret-telemeter-client\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376154 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376192 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-registration-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376224 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m49gk\" (UniqueName: \"kubernetes.io/projected/2b94c3af-6055-4f33-bd4b-57d4411d0b28-kube-api-access-m49gk\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376247 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376266 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-systemd\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376287 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376303 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-cni-bin\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376320 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5tj\" (UniqueName: \"kubernetes.io/projected/6a763f73-62ad-43ed-babf-fb2ced67616a-kube-api-access-hn5tj\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376336 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysconfig\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376354 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-host\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376371 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-hosts-file\") pod \"node-resolver-q8z8s\" (UID: \"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6\") " pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376388 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82405fd-bde5-4428-8b23-2be99042feef-metrics-certs\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376388 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56x5w\" (UniqueName: \"kubernetes.io/projected/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-kube-api-access-56x5w\") pod \"node-resolver-q8z8s\" (UID: \"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6\") " pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376423 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376440 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376457 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-system-cni-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376474 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f76551a8-8b0b-459d-86f5-84cf822e83b8-tmp-dir\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376491 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b-konnectivity-ca\") pod \"konnectivity-agent-hsqk7\" (UID: \"8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b\") " pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376506 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-run\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376524 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j75nb\" (UniqueName: \"kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-kube-api-access-j75nb\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376551 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.376806 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376720 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f76551a8-8b0b-459d-86f5-84cf822e83b8-tmp-dir\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376901 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-multus-certs\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376942 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-kubernetes\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376965 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-out\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376976 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b-konnectivity-ca\") pod \"konnectivity-agent-hsqk7\" (UID: \"8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b\") " pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.376982 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-kubelet\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377058 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-out\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377101 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-accelerators-collector-config\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377133 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/866d9622-0c68-4e07-94ef-94cabbd34f86-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377176 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c94rp\" (UniqueName: \"kubernetes.io/projected/3e6de072-eb99-48f0-ade6-f7fad4be241b-kube-api-access-c94rp\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377251 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c032d4ff-047f-43a9-a27e-41affb60615d-serviceca\") pod \"node-ca-mnq7z\" (UID: \"c032d4ff-047f-43a9-a27e-41affb60615d\") " pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377280 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysctl-conf\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377309 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377337 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377374 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377401 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovn-node-metrics-cert\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377427 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c032d4ff-047f-43a9-a27e-41affb60615d-host\") pod \"node-ca-mnq7z\" (UID: \"c032d4ff-047f-43a9-a27e-41affb60615d\") " pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377449 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377496 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx2kb\" (UniqueName: \"kubernetes.io/projected/c032d4ff-047f-43a9-a27e-41affb60615d-kube-api-access-fx2kb\") pod \"node-ca-mnq7z\" (UID: \"c032d4ff-047f-43a9-a27e-41affb60615d\") " pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377537 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqch6\" (UniqueName: \"kubernetes.io/projected/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-kube-api-access-fqch6\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377570 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377626 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l75cz\" (UniqueName: \"kubernetes.io/projected/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-api-access-l75cz\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:43.377683 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377671 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377706 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377734 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-web-config\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377763 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f64a1d53-38a0-4133-accb-411e2dc6fc64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377842 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377863 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c032d4ff-047f-43a9-a27e-41affb60615d-serviceca\") pod \"node-ca-mnq7z\" (UID: \"c032d4ff-047f-43a9-a27e-41affb60615d\") " pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377876 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-etc-selinux\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377922 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377954 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.377981 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f76551a8-8b0b-459d-86f5-84cf822e83b8-metrics-tls\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.378006 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-sys\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.378033 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-cni-multus\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.378073 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-var-lib-kubelet\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.378132 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.378156 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-cni-binary-copy\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.378172 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-env-overrides\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.378191 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-wtmp\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.378207 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysctl-d\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.378226 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-tls\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.378284 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:43.378507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.378314 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-modprobe-d\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.379187 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.378660 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 08:54:43.379187 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.379017 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 08:54:43.379187 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.379026 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 08:54:43.379187 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.379033 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 08:54:43.379390 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.379375 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:54:43.379390 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.379386 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:54:43.379479 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.379406 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 08:54:43.379479 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.379433 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 08:54:43.379655 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.379640 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 08:54:43.379702 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.379680 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 08:54:43.379833 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.379820 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 08:54:43.379877 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.379862 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 08:54:43.379923 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.379891 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 08:54:43.380345 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380315 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 08:54:43.380345 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380316 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 08:54:43.380892 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380434 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 08:54:43.380892 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380436 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 08:54:43.380892 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380485 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 08:54:43.380892 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380495 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 08:54:43.380892 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380498 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 08:54:43.380892 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380486 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 08:54:43.380892 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380520 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 08:54:43.380892 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380552 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 08:54:43.380892 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380648 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 08:54:43.380892 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380707 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 08:54:43.380892 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380716 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 08:54:43.380892 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380758 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 08:54:43.380892 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380876 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 08:54:43.380892 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380890 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 08:54:43.381622 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380934 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 08:54:43.381622 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380945 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 08:54:43.381622 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380945 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 08:54:43.381622 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380958 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 08:54:43.381622 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.380987 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 08:54:43.381622 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.381203 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 08:54:43.381622 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.381238 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 08:54:43.381622 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.381286 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 08:54:43.381622 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.381357 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 08:54:43.381622 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.381369 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 08:54:43.381622 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.381406 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 08:54:43.382011 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.381787 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 08:54:43.382011 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.381879 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 08:54:43.382011 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.381881 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 08:54:43.382011 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.381899 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 08:54:43.382011 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.381899 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 08:54:43.382011 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.381944 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jptkp\"" Oct 30 08:54:43.383144 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.383118 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-web-config\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.383237 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.383225 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.383294 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.383234 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.383294 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.383246 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ecaa283-05a2-4f68-857e-dd14c857c804-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:43.383294 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.383262 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovnkube-config\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.383438 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.383376 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.384169 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.384155 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a763f73-62ad-43ed-babf-fb2ced67616a-metrics-client-ca\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.384206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.384161 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/866d9622-0c68-4e07-94ef-94cabbd34f86-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:43.384236 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.384212 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-cni-binary-copy\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.384316 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.384300 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.384316 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.384310 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/87f93427-45fe-46a3-8b9f-23d865534d41-iptables-alerter-script\") pod \"iptables-alerter-kft2v\" (UID: \"87f93427-45fe-46a3-8b9f-23d865534d41\") " pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:54:43.384382 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.384359 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-volume\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.385365 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.385350 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:43.385414 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.385364 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-daemon-config\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.385414 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.385372 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f76551a8-8b0b-459d-86f5-84cf822e83b8-config-volume\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:43.385414 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.385394 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/866d9622-0c68-4e07-94ef-94cabbd34f86-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:43.385539 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.385480 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e23f66da-2587-4ce8-bb3a-a344e5dad910-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:43.385539 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.385525 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2eebeab0-d5a7-443d-9ed8-365761816c98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.385670 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.385658 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-serving-certs-ca-bundle\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:43.385805 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.385792 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.385914 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.385902 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.385993 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.385981 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.386109 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.386098 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-metrics-client-ca\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.386161 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.386149 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.386199 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.386178 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-metrics-client-ca\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:43.386270 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.386258 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f64a1d53-38a0-4133-accb-411e2dc6fc64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:43.386306 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.386274 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e23f66da-2587-4ce8-bb3a-a344e5dad910-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:43.387748 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.387730 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.387811 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.387770 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.387811 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.387774 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.387895 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.387828 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-accelerators-collector-config\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.387895 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.387883 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.387961 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.387938 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-web-config\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.389089 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.389072 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.389131 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.389074 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-cni-binary-copy\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.389131 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.389098 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.389202 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.389163 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-env-overrides\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.392836 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.392815 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 08:54:43.392836 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.392819 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 08:54:43.392836 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.392837 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 08:54:43.392836 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.392846 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 08:54:43.392836 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.392878 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 08:54:43.393147 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.392879 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 08:54:43.393147 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.392983 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 08:54:43.393147 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393025 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 08:54:43.393147 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393040 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 08:54:43.393147 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393063 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qz7rt\"" Oct 30 08:54:43.393147 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393063 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 08:54:43.393147 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393067 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 08:54:43.393557 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393429 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3a6is943ja23r\"" Oct 30 08:54:43.393557 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393444 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5ptj4\"" Oct 30 08:54:43.393557 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393513 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kd468\"" Oct 30 08:54:43.393978 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393623 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 08:54:43.393978 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393682 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 08:54:43.393978 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393819 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 08:54:43.393978 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393835 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gbg4x\"" Oct 30 08:54:43.393978 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393845 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2jshw\"" Oct 30 08:54:43.393978 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393873 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7pnl7\"" Oct 30 08:54:43.393978 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393883 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xqvv4\"" Oct 30 08:54:43.393978 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393876 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 08:54:43.393978 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393901 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cnfjl\"" Oct 30 08:54:43.393978 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393915 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.393978 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393939 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-6sdrk\"" Oct 30 08:54:43.393978 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393985 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-752kq\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393989 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.393990 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394094 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394221 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394298 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394380 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394487 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-tls\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394518 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394656 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394720 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4gqanurtnecfq\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394741 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-twrgj\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394764 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394804 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovnkube-script-lib\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394818 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394819 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4xr7r\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394853 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wdt5m\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394889 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cb5mw\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394898 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.394943 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wqhvd\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.395040 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.395052 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.395044 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-federate-client-tls\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:43.395133 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.395110 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hr7xt\"" Oct 30 08:54:43.396264 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.395151 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 08:54:43.396264 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.395155 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.396264 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.395212 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tpklt\"" Oct 30 08:54:43.396264 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.395357 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 08:54:43.396264 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.395505 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 08:54:43.396264 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.395632 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:43.396264 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.395886 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-grpc-tls\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.396264 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.396142 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.396559 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.396351 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-secret-telemeter-client\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:43.396559 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.396453 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7400965-54de-4420-a62a-37baf38d949d-original-pull-secret\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:43.397218 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.397199 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.397278 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.397244 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.397428 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.397407 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/866d9622-0c68-4e07-94ef-94cabbd34f86-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:43.397839 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.397607 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.397839 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.397629 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.397839 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.397737 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b94c3af-6055-4f33-bd4b-57d4411d0b28-ovn-node-metrics-cert\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.397839 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.397792 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.398035 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.397984 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.398035 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.397996 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f64a1d53-38a0-4133-accb-411e2dc6fc64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:43.398457 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.398397 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f76551a8-8b0b-459d-86f5-84cf822e83b8-metrics-tls\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:43.398457 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.398432 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:43.398560 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.398512 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-tls\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.398560 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.398522 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:43.403474 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.403451 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.403546 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.403517 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3e6de072-eb99-48f0-ade6-f7fad4be241b-telemeter-client-tls\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:43.403658 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.403640 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.405343 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.405325 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab3d086c-96af-4210-abf6-0cfbfe3afb20-cert\") pod \"ingress-canary-kwlz8\" (UID: \"ab3d086c-96af-4210-abf6-0cfbfe3afb20\") " pod="openshift-ingress-canary/ingress-canary-kwlz8" Oct 30 08:54:43.405343 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.405333 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e23f66da-2587-4ce8-bb3a-a344e5dad910-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:43.406993 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.406974 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 08:54:43.406993 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.406978 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 08:54:43.407137 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.406994 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 08:54:43.407137 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.407082 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 08:54:43.407514 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.407496 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hj9q\" (UniqueName: \"kubernetes.io/projected/f76551a8-8b0b-459d-86f5-84cf822e83b8-kube-api-access-2hj9q\") pod \"dns-default-fxq45\" (UID: \"f76551a8-8b0b-459d-86f5-84cf822e83b8\") " pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:43.407602 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.407517 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgcgt\" (UniqueName: \"kubernetes.io/projected/f64a1d53-38a0-4133-accb-411e2dc6fc64-kube-api-access-tgcgt\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:43.407712 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.407683 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgsqg\" (UniqueName: \"kubernetes.io/projected/87f93427-45fe-46a3-8b9f-23d865534d41-kube-api-access-kgsqg\") pod \"iptables-alerter-kft2v\" (UID: \"87f93427-45fe-46a3-8b9f-23d865534d41\") " pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:54:43.407779 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.407723 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56x5w\" (UniqueName: \"kubernetes.io/projected/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-kube-api-access-56x5w\") pod \"node-resolver-q8z8s\" (UID: \"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6\") " pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:54:43.407847 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.407788 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frlr5\" (UniqueName: \"kubernetes.io/projected/e23f66da-2587-4ce8-bb3a-a344e5dad910-kube-api-access-frlr5\") pod \"prometheus-operator-9cd89b74c-z42cw\" (UID: \"e23f66da-2587-4ce8-bb3a-a344e5dad910\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z42cw" Oct 30 08:54:43.408009 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.407992 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5tj\" (UniqueName: \"kubernetes.io/projected/6a763f73-62ad-43ed-babf-fb2ced67616a-kube-api-access-hn5tj\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.408069 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.408016 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmq2k\" (UniqueName: \"kubernetes.io/projected/f82405fd-bde5-4428-8b23-2be99042feef-kube-api-access-bmq2k\") pod \"network-metrics-daemon-ktt7j\" (UID: \"f82405fd-bde5-4428-8b23-2be99042feef\") " pod="openshift-multus/network-metrics-daemon-ktt7j" Oct 30 08:54:43.408158 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.408136 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j75nb\" (UniqueName: \"kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-kube-api-access-j75nb\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.408280 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.408258 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m49gk\" (UniqueName: \"kubernetes.io/projected/2b94c3af-6055-4f33-bd4b-57d4411d0b28-kube-api-access-m49gk\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.408347 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.408266 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6de072-eb99-48f0-ade6-f7fad4be241b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:43.408347 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.408297 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psn67\" (UniqueName: \"kubernetes.io/projected/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-kube-api-access-psn67\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.408870 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.408427 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkdr4\" (UniqueName: \"kubernetes.io/projected/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-kube-api-access-vkdr4\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.408870 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.408495 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx5wx\" (UniqueName: \"kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-kube-api-access-tx5wx\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.408870 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.408535 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c94rp\" (UniqueName: \"kubernetes.io/projected/3e6de072-eb99-48f0-ade6-f7fad4be241b-kube-api-access-c94rp\") pod \"telemeter-client-65c85d64c4-d8zwz\" (UID: \"3e6de072-eb99-48f0-ade6-f7fad4be241b\") " pod="openshift-monitoring/telemeter-client-65c85d64c4-d8zwz" Oct 30 08:54:43.409049 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.408881 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx2kb\" (UniqueName: \"kubernetes.io/projected/c032d4ff-047f-43a9-a27e-41affb60615d-kube-api-access-fx2kb\") pod \"node-ca-mnq7z\" (UID: \"c032d4ff-047f-43a9-a27e-41affb60615d\") " pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:54:43.409049 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.408989 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqch6\" (UniqueName: \"kubernetes.io/projected/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-kube-api-access-fqch6\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.409049 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.408994 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9984j\" (UniqueName: \"kubernetes.io/projected/7c5d7c66-ce15-4d20-bc37-4d832f308d9f-kube-api-access-9984j\") pod \"thanos-querier-5f48c4fbf4-lhblx\" (UID: \"7c5d7c66-ce15-4d20-bc37-4d832f308d9f\") " pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:43.409049 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.409021 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgbtw\" (UniqueName: \"kubernetes.io/projected/2eebeab0-d5a7-443d-9ed8-365761816c98-kube-api-access-wgbtw\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.409049 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.409037 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szrjj\" (UniqueName: \"kubernetes.io/projected/25cb63c1-5df3-4be8-b407-b9bf2641cf95-kube-api-access-szrjj\") pod \"network-check-target-tf9r8\" (UID: \"25cb63c1-5df3-4be8-b407-b9bf2641cf95\") " pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:54:43.409308 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.409052 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l75cz\" (UniqueName: \"kubernetes.io/projected/1ecaa283-05a2-4f68-857e-dd14c857c804-kube-api-access-l75cz\") pod \"kube-state-metrics-5dc8d7d89b-9tlbg\" (UID: \"1ecaa283-05a2-4f68-857e-dd14c857c804\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9tlbg" Oct 30 08:54:43.409308 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.409183 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjskv\" (UniqueName: \"kubernetes.io/projected/ab3d086c-96af-4210-abf6-0cfbfe3afb20-kube-api-access-sjskv\") pod \"ingress-canary-kwlz8\" (UID: \"ab3d086c-96af-4210-abf6-0cfbfe3afb20\") " pod="openshift-ingress-canary/ingress-canary-kwlz8" Oct 30 08:54:43.409545 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.409526 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvg8d\" (UniqueName: \"kubernetes.io/projected/866d9622-0c68-4e07-94ef-94cabbd34f86-kube-api-access-bvg8d\") pod \"openshift-state-metrics-7bd7489957-cddd2\" (UID: \"866d9622-0c68-4e07-94ef-94cabbd34f86\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cddd2" Oct 30 08:54:43.413980 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.413960 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:54:43.414736 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.414720 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.417331 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.417315 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:43.478845 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.478793 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-systemd\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.478845 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.478823 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-ovn\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.478845 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.478840 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-cni-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.478974 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.478845 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-systemd\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.478974 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.478880 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-ovn\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.478974 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.478853 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-cni-bin\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.478974 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.478893 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-cni-bin\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.478974 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.478911 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-cnibin\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.478974 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.478939 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-socket-dir-parent\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.478974 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.478941 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-cni-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.478974 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.478957 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.478974 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.478972 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-k8s-cni-cncf-io\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.478982 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-cnibin\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479007 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-run-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479011 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-k8s-cni-cncf-io\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479017 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7400965-54de-4420-a62a-37baf38d949d-kubelet-config\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479033 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-socket-dir-parent\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479050 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7400965-54de-4420-a62a-37baf38d949d-kubelet-config\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479042 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-root\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479059 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-root\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479090 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-systemd-units\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479113 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-systemd-units\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479118 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479147 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-cnibin\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479171 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87f93427-45fe-46a3-8b9f-23d865534d41-host-slash\") pod \"iptables-alerter-kft2v\" (UID: \"87f93427-45fe-46a3-8b9f-23d865534d41\") " pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479198 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-etc-kubernetes\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479199 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479224 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-system-cni-dir\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479232 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-cnibin\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479239 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-etc-kubernetes\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479249 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-hostroot\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479259 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87f93427-45fe-46a3-8b9f-23d865534d41-host-slash\") pod \"iptables-alerter-kft2v\" (UID: \"87f93427-45fe-46a3-8b9f-23d865534d41\") " pod="openshift-network-operator/iptables-alerter-kft2v" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479290 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-run-netns\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479295 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-system-cni-dir\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479316 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-hostroot\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479327 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-run-netns\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479359 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-conf-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479373 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-node-log\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479396 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479413 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-multus-conf-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479419 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-node-log\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479420 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-var-lib-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479438 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479445 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-var-lib-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479451 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-log-socket\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479472 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f64a1d53-38a0-4133-accb-411e2dc6fc64-crio-socket\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479491 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-slash\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479505 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7400965-54de-4420-a62a-37baf38d949d-dbus\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479515 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-log-socket\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479561 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f64a1d53-38a0-4133-accb-411e2dc6fc64-crio-socket\") pod \"insights-runtime-extractor-vbz6m\" (UID: \"f64a1d53-38a0-4133-accb-411e2dc6fc64\") " pod="openshift-insights/insights-runtime-extractor-vbz6m" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479617 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-registration-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479630 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-slash\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479647 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-systemd\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479674 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-cni-bin\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479689 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7400965-54de-4420-a62a-37baf38d949d-dbus\") pod \"global-pull-secret-syncer-m5sz6\" (UID: \"c7400965-54de-4420-a62a-37baf38d949d\") " pod="kube-system/global-pull-secret-syncer-m5sz6" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479709 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-registration-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479725 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-cni-bin\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479728 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-systemd\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479765 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysconfig\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479783 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-host\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479797 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-hosts-file\") pod \"node-resolver-q8z8s\" (UID: \"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6\") " pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479811 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-system-cni-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479824 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysconfig\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479839 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-run\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479855 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-multus-certs\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479869 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-kubernetes\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479864 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-host\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479880 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6-hosts-file\") pod \"node-resolver-q8z8s\" (UID: \"b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6\") " pod="openshift-dns/node-resolver-q8z8s" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479893 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-kubelet\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479896 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-system-cni-dir\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479908 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-run\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479916 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-multus-certs\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479929 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-kubernetes\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.479927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479918 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysctl-conf\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479958 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c032d4ff-047f-43a9-a27e-41affb60615d-host\") pod \"node-ca-mnq7z\" (UID: \"c032d4ff-047f-43a9-a27e-41affb60615d\") " pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479938 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-kubelet\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479980 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.479983 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c032d4ff-047f-43a9-a27e-41affb60615d-host\") pod \"node-ca-mnq7z\" (UID: \"c032d4ff-047f-43a9-a27e-41affb60615d\") " pod="openshift-image-registry/node-ca-mnq7z" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480020 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2eebeab0-d5a7-443d-9ed8-365761816c98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwg67\" (UID: \"2eebeab0-d5a7-443d-9ed8-365761816c98\") " pod="openshift-multus/multus-additional-cni-plugins-hwg67" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480031 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-etc-selinux\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480048 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysctl-conf\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480060 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-sys\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480072 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-etc-selinux\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480086 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-cni-multus\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480109 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-var-lib-kubelet\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480133 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-wtmp\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480147 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-cni-multus\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480157 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysctl-d\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480157 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-sys\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480175 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-var-lib-kubelet\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480185 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-node-exporter-wtmp\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480198 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-modprobe-d\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480225 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-etc-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480249 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-run-ovn-kubernetes\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480255 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-sysctl-d\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480272 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-socket-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480284 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-etc-openvswitch\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480284 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-etc-modprobe-d\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480297 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-cni-netd\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480315 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-run-ovn-kubernetes\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480323 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-sys\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480330 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-socket-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480334 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b94c3af-6055-4f33-bd4b-57d4411d0b28-host-cni-netd\") pod \"ovnkube-node-mf5b5\" (UID: \"2b94c3af-6055-4f33-bd4b-57d4411d0b28\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480349 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-lib-modules\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480361 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a763f73-62ad-43ed-babf-fb2ced67616a-sys\") pod \"node-exporter-8kgvx\" (UID: \"6a763f73-62ad-43ed-babf-fb2ced67616a\") " pod="openshift-monitoring/node-exporter-8kgvx" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480372 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-sys-fs\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480398 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-netns\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480438 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-kubelet\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480452 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-sys-fs\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480451 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-run-netns\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480460 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-device-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480483 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20f4074-b1a5-4116-ac25-2d8d4235b7b7-host-var-lib-kubelet\") pod \"multus-qgqjn\" (UID: \"d20f4074-b1a5-4116-ac25-2d8d4235b7b7\") " pod="openshift-multus/multus-qgqjn" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480465 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dee6e4b-1bc3-4718-a1a6-d268dd6b405c-lib-modules\") pod \"tuned-nt944\" (UID: \"7dee6e4b-1bc3-4718-a1a6-d268dd6b405c\") " pod="openshift-cluster-node-tuning-operator/tuned-nt944" Oct 30 08:54:43.481289 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.480488 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/70cee2c6-bc37-4ab5-9f5f-5ded85c0e334-device-dir\") pod \"aws-ebs-csi-driver-node-n4h5p\" (UID: \"70cee2c6-bc37-4ab5-9f5f-5ded85c0e334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n4h5p" Oct 30 08:54:43.828188 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:43.828123 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:44.574986 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:44.574958 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:45.328908 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:45.328879 5938 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 08:54:45.769644 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:45.769609 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:45.785911 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:45.785887 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:54:45.807208 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:54:45.807178 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 08:54:45.917519 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:45.917473 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fxq45" Oct 30 08:54:48.411165 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:48.411132 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f48c4fbf4-lhblx" Oct 30 08:54:49.909875 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:49.909843 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf5b5" Oct 30 08:54:52.266322 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:54:52.266255 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:55:11.666003 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:11.665973 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hsqk7" Oct 30 08:55:12.269297 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:12.269264 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tf9r8" Oct 30 08:55:13.861982 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:13.858942 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 08:55:13.861982 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:13.859654 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="alertmanager" containerID="cri-o://827272fdb9b7e83447fd6d35d96c4726bc1f60d78fd3d96158aacabc56c387f2" gracePeriod=120 Oct 30 08:55:13.861982 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:13.859993 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="kube-rbac-proxy" containerID="cri-o://e586f4097d1677601287d5f8ab7023f50417fe891c9c2b6f85eae471ce880aec" gracePeriod=120 Oct 30 08:55:13.861982 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:13.860105 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="kube-rbac-proxy-web" containerID="cri-o://16d53eaf6744f3c5571afdd113bfb07aea1a8e4d0fe80f1df510e6b77e0c168d" gracePeriod=120 Oct 30 08:55:13.861982 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:13.860161 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="prom-label-proxy" containerID="cri-o://e7945184e2389a6ef4ffb71f3ac48af885e17a22340468496b63ca8ae5ab1dde" gracePeriod=120 Oct 30 08:55:13.861982 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:13.860228 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="config-reloader" containerID="cri-o://fdf66999accbbe1ad4c116bb62c82a6764715ed865aa7c9091575836470a4434" gracePeriod=120 Oct 30 08:55:13.861982 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:13.860262 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="kube-rbac-proxy-metric" containerID="cri-o://e95aa4a98351e9569f8478ccac710143820e3866cd01b81f1aecb8a9f09eb8a4" gracePeriod=120 Oct 30 08:55:14.434173 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:14.434144 5938 generic.go:356] "Generic (PLEG): container finished" podID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerID="e7945184e2389a6ef4ffb71f3ac48af885e17a22340468496b63ca8ae5ab1dde" exitCode=0 Oct 30 08:55:14.434173 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:14.434167 5938 generic.go:356] "Generic (PLEG): container finished" podID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerID="e95aa4a98351e9569f8478ccac710143820e3866cd01b81f1aecb8a9f09eb8a4" exitCode=0 Oct 30 08:55:14.434173 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:14.434177 5938 generic.go:356] "Generic (PLEG): container finished" podID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerID="e586f4097d1677601287d5f8ab7023f50417fe891c9c2b6f85eae471ce880aec" exitCode=0 Oct 30 08:55:14.434173 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:14.434182 5938 generic.go:356] "Generic (PLEG): container finished" podID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerID="fdf66999accbbe1ad4c116bb62c82a6764715ed865aa7c9091575836470a4434" exitCode=0 Oct 30 08:55:14.434173 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:14.434188 5938 generic.go:356] "Generic (PLEG): container finished" podID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerID="827272fdb9b7e83447fd6d35d96c4726bc1f60d78fd3d96158aacabc56c387f2" exitCode=0 Oct 30 08:55:14.434461 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:14.434208 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0ac551a-4850-4c58-9fe8-6432e5d3301c","Type":"ContainerDied","Data":"e7945184e2389a6ef4ffb71f3ac48af885e17a22340468496b63ca8ae5ab1dde"} Oct 30 08:55:14.434461 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:14.434269 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0ac551a-4850-4c58-9fe8-6432e5d3301c","Type":"ContainerDied","Data":"e95aa4a98351e9569f8478ccac710143820e3866cd01b81f1aecb8a9f09eb8a4"} Oct 30 08:55:14.434461 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:14.434310 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0ac551a-4850-4c58-9fe8-6432e5d3301c","Type":"ContainerDied","Data":"e586f4097d1677601287d5f8ab7023f50417fe891c9c2b6f85eae471ce880aec"} Oct 30 08:55:14.434461 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:14.434334 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0ac551a-4850-4c58-9fe8-6432e5d3301c","Type":"ContainerDied","Data":"fdf66999accbbe1ad4c116bb62c82a6764715ed865aa7c9091575836470a4434"} Oct 30 08:55:14.434461 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:14.434359 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0ac551a-4850-4c58-9fe8-6432e5d3301c","Type":"ContainerDied","Data":"827272fdb9b7e83447fd6d35d96c4726bc1f60d78fd3d96158aacabc56c387f2"} Oct 30 08:55:15.110076 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.110052 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.226985 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.226916 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-volume\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-volume\") pod \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " Oct 30 08:55:15.226985 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.226966 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-cluster-tls-config\") pod \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " Oct 30 08:55:15.227191 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.226995 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-kube-api-access-j75nb\" (UniqueName: \"kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-kube-api-access-j75nb\") pod \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " Oct 30 08:55:15.227191 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.227027 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-web\") pod \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " Oct 30 08:55:15.227191 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.227087 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-metrics-client-ca\") pod \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " Oct 30 08:55:15.227191 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.227123 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " Oct 30 08:55:15.227191 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.227167 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-trusted-ca-bundle\") pod \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " Oct 30 08:55:15.227444 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.227243 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-main-db\") pod \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " Oct 30 08:55:15.227444 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.227294 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-out\") pod \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " Oct 30 08:55:15.227444 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.227322 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-web-config\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-web-config\") pod \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " Oct 30 08:55:15.227444 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.227352 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-tls-assets\" (UniqueName: \"kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-tls-assets\") pod \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " Oct 30 08:55:15.227444 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.227394 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-main-tls\") pod \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " Oct 30 08:55:15.227444 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.227424 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy\") pod \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\" (UID: \"b0ac551a-4850-4c58-9fe8-6432e5d3301c\") " Oct 30 08:55:15.227800 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.227713 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-metrics-client-ca" pod "b0ac551a-4850-4c58-9fe8-6432e5d3301c" (UID: "b0ac551a-4850-4c58-9fe8-6432e5d3301c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:15.228042 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.228011 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-trusted-ca-bundle" pod "b0ac551a-4850-4c58-9fe8-6432e5d3301c" (UID: "b0ac551a-4850-4c58-9fe8-6432e5d3301c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:15.229348 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.229312 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-main-db" pod "b0ac551a-4850-4c58-9fe8-6432e5d3301c" (UID: "b0ac551a-4850-4c58-9fe8-6432e5d3301c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:55:15.230669 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.230622 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-out" pod "b0ac551a-4850-4c58-9fe8-6432e5d3301c" (UID: "b0ac551a-4850-4c58-9fe8-6432e5d3301c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:55:15.231556 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.231527 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-kube-api-access-j75nb" pod "b0ac551a-4850-4c58-9fe8-6432e5d3301c" (UID: "b0ac551a-4850-4c58-9fe8-6432e5d3301c"). InnerVolumeSpecName "kube-api-access-j75nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:55:15.232695 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.232656 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-tls-assets" pod "b0ac551a-4850-4c58-9fe8-6432e5d3301c" (UID: "b0ac551a-4850-4c58-9fe8-6432e5d3301c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:55:15.232793 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.232710 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy" pod "b0ac551a-4850-4c58-9fe8-6432e5d3301c" (UID: "b0ac551a-4850-4c58-9fe8-6432e5d3301c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:15.232893 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.232870 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-web" pod "b0ac551a-4850-4c58-9fe8-6432e5d3301c" (UID: "b0ac551a-4850-4c58-9fe8-6432e5d3301c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:15.233027 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.233009 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-volume" pod "b0ac551a-4850-4c58-9fe8-6432e5d3301c" (UID: "b0ac551a-4850-4c58-9fe8-6432e5d3301c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:15.233251 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.233231 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-metric" pod "b0ac551a-4850-4c58-9fe8-6432e5d3301c" (UID: "b0ac551a-4850-4c58-9fe8-6432e5d3301c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:15.233714 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.233691 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-main-tls" pod "b0ac551a-4850-4c58-9fe8-6432e5d3301c" (UID: "b0ac551a-4850-4c58-9fe8-6432e5d3301c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:15.236645 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.236622 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-cluster-tls-config" pod "b0ac551a-4850-4c58-9fe8-6432e5d3301c" (UID: "b0ac551a-4850-4c58-9fe8-6432e5d3301c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:15.244977 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.244950 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-web-config" pod "b0ac551a-4850-4c58-9fe8-6432e5d3301c" (UID: "b0ac551a-4850-4c58-9fe8-6432e5d3301c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:15.328029 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.328002 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-metrics-client-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:15.328029 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.328029 5938 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:15.328213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.328043 5938 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:15.328213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.328058 5938 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-alertmanager-main-db\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:15.328213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.328070 5938 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-out\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:15.328213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.328081 5938 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-web-config\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:15.328213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.328092 5938 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-tls-assets\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:15.328213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.328102 5938 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-main-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:15.328213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.328115 5938 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:15.328213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.328127 5938 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:15.328213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.328141 5938 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-cluster-tls-config\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:15.328213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.328153 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j75nb\" (UniqueName: \"kubernetes.io/projected/b0ac551a-4850-4c58-9fe8-6432e5d3301c-kube-api-access-j75nb\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:15.328213 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.328166 5938 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0ac551a-4850-4c58-9fe8-6432e5d3301c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:15.441142 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.441114 5938 generic.go:356] "Generic (PLEG): container finished" podID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerID="16d53eaf6744f3c5571afdd113bfb07aea1a8e4d0fe80f1df510e6b77e0c168d" exitCode=0 Oct 30 08:55:15.441281 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.441195 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.441281 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.441199 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0ac551a-4850-4c58-9fe8-6432e5d3301c","Type":"ContainerDied","Data":"16d53eaf6744f3c5571afdd113bfb07aea1a8e4d0fe80f1df510e6b77e0c168d"} Oct 30 08:55:15.441281 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.441271 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0ac551a-4850-4c58-9fe8-6432e5d3301c","Type":"ContainerDied","Data":"f6ee89a073f2369082eb1f51a558980516571acf01d445de228953a6a7804334"} Oct 30 08:55:15.441408 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.441306 5938 scope.go:122] "RemoveContainer" containerID="e7945184e2389a6ef4ffb71f3ac48af885e17a22340468496b63ca8ae5ab1dde" Oct 30 08:55:15.450680 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.450657 5938 scope.go:122] "RemoveContainer" containerID="e95aa4a98351e9569f8478ccac710143820e3866cd01b81f1aecb8a9f09eb8a4" Oct 30 08:55:15.457465 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.457448 5938 scope.go:122] "RemoveContainer" containerID="e586f4097d1677601287d5f8ab7023f50417fe891c9c2b6f85eae471ce880aec" Oct 30 08:55:15.464222 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.464209 5938 scope.go:122] "RemoveContainer" containerID="16d53eaf6744f3c5571afdd113bfb07aea1a8e4d0fe80f1df510e6b77e0c168d" Oct 30 08:55:15.470528 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.470512 5938 scope.go:122] "RemoveContainer" containerID="fdf66999accbbe1ad4c116bb62c82a6764715ed865aa7c9091575836470a4434" Oct 30 08:55:15.477233 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.477217 5938 scope.go:122] "RemoveContainer" containerID="827272fdb9b7e83447fd6d35d96c4726bc1f60d78fd3d96158aacabc56c387f2" Oct 30 08:55:15.483947 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.483929 5938 scope.go:122] "RemoveContainer" containerID="428a41797386d649baf498c2ec11969043e66d787e4e260eb3d334cd7fc51391" Oct 30 08:55:15.490515 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.490496 5938 scope.go:122] "RemoveContainer" containerID="e7945184e2389a6ef4ffb71f3ac48af885e17a22340468496b63ca8ae5ab1dde" Oct 30 08:55:15.490803 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:55:15.490785 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7945184e2389a6ef4ffb71f3ac48af885e17a22340468496b63ca8ae5ab1dde\": container with ID starting with e7945184e2389a6ef4ffb71f3ac48af885e17a22340468496b63ca8ae5ab1dde not found: ID does not exist" containerID="e7945184e2389a6ef4ffb71f3ac48af885e17a22340468496b63ca8ae5ab1dde" Oct 30 08:55:15.490851 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.490811 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7945184e2389a6ef4ffb71f3ac48af885e17a22340468496b63ca8ae5ab1dde"} err="failed to get container status \"e7945184e2389a6ef4ffb71f3ac48af885e17a22340468496b63ca8ae5ab1dde\": rpc error: code = NotFound desc = could not find container \"e7945184e2389a6ef4ffb71f3ac48af885e17a22340468496b63ca8ae5ab1dde\": container with ID starting with e7945184e2389a6ef4ffb71f3ac48af885e17a22340468496b63ca8ae5ab1dde not found: ID does not exist" Oct 30 08:55:15.490851 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.490840 5938 scope.go:122] "RemoveContainer" containerID="e95aa4a98351e9569f8478ccac710143820e3866cd01b81f1aecb8a9f09eb8a4" Oct 30 08:55:15.491072 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:55:15.491055 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95aa4a98351e9569f8478ccac710143820e3866cd01b81f1aecb8a9f09eb8a4\": container with ID starting with e95aa4a98351e9569f8478ccac710143820e3866cd01b81f1aecb8a9f09eb8a4 not found: ID does not exist" containerID="e95aa4a98351e9569f8478ccac710143820e3866cd01b81f1aecb8a9f09eb8a4" Oct 30 08:55:15.491135 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.491081 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95aa4a98351e9569f8478ccac710143820e3866cd01b81f1aecb8a9f09eb8a4"} err="failed to get container status \"e95aa4a98351e9569f8478ccac710143820e3866cd01b81f1aecb8a9f09eb8a4\": rpc error: code = NotFound desc = could not find container \"e95aa4a98351e9569f8478ccac710143820e3866cd01b81f1aecb8a9f09eb8a4\": container with ID starting with e95aa4a98351e9569f8478ccac710143820e3866cd01b81f1aecb8a9f09eb8a4 not found: ID does not exist" Oct 30 08:55:15.491135 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.491103 5938 scope.go:122] "RemoveContainer" containerID="e586f4097d1677601287d5f8ab7023f50417fe891c9c2b6f85eae471ce880aec" Oct 30 08:55:15.491344 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:55:15.491327 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e586f4097d1677601287d5f8ab7023f50417fe891c9c2b6f85eae471ce880aec\": container with ID starting with e586f4097d1677601287d5f8ab7023f50417fe891c9c2b6f85eae471ce880aec not found: ID does not exist" containerID="e586f4097d1677601287d5f8ab7023f50417fe891c9c2b6f85eae471ce880aec" Oct 30 08:55:15.491378 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.491350 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e586f4097d1677601287d5f8ab7023f50417fe891c9c2b6f85eae471ce880aec"} err="failed to get container status \"e586f4097d1677601287d5f8ab7023f50417fe891c9c2b6f85eae471ce880aec\": rpc error: code = NotFound desc = could not find container \"e586f4097d1677601287d5f8ab7023f50417fe891c9c2b6f85eae471ce880aec\": container with ID starting with e586f4097d1677601287d5f8ab7023f50417fe891c9c2b6f85eae471ce880aec not found: ID does not exist" Oct 30 08:55:15.491378 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.491364 5938 scope.go:122] "RemoveContainer" containerID="16d53eaf6744f3c5571afdd113bfb07aea1a8e4d0fe80f1df510e6b77e0c168d" Oct 30 08:55:15.491632 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:55:15.491611 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d53eaf6744f3c5571afdd113bfb07aea1a8e4d0fe80f1df510e6b77e0c168d\": container with ID starting with 16d53eaf6744f3c5571afdd113bfb07aea1a8e4d0fe80f1df510e6b77e0c168d not found: ID does not exist" containerID="16d53eaf6744f3c5571afdd113bfb07aea1a8e4d0fe80f1df510e6b77e0c168d" Oct 30 08:55:15.491679 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.491639 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d53eaf6744f3c5571afdd113bfb07aea1a8e4d0fe80f1df510e6b77e0c168d"} err="failed to get container status \"16d53eaf6744f3c5571afdd113bfb07aea1a8e4d0fe80f1df510e6b77e0c168d\": rpc error: code = NotFound desc = could not find container \"16d53eaf6744f3c5571afdd113bfb07aea1a8e4d0fe80f1df510e6b77e0c168d\": container with ID starting with 16d53eaf6744f3c5571afdd113bfb07aea1a8e4d0fe80f1df510e6b77e0c168d not found: ID does not exist" Oct 30 08:55:15.491679 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.491652 5938 scope.go:122] "RemoveContainer" containerID="fdf66999accbbe1ad4c116bb62c82a6764715ed865aa7c9091575836470a4434" Oct 30 08:55:15.491917 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:55:15.491903 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf66999accbbe1ad4c116bb62c82a6764715ed865aa7c9091575836470a4434\": container with ID starting with fdf66999accbbe1ad4c116bb62c82a6764715ed865aa7c9091575836470a4434 not found: ID does not exist" containerID="fdf66999accbbe1ad4c116bb62c82a6764715ed865aa7c9091575836470a4434" Oct 30 08:55:15.491963 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.491920 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf66999accbbe1ad4c116bb62c82a6764715ed865aa7c9091575836470a4434"} err="failed to get container status \"fdf66999accbbe1ad4c116bb62c82a6764715ed865aa7c9091575836470a4434\": rpc error: code = NotFound desc = could not find container \"fdf66999accbbe1ad4c116bb62c82a6764715ed865aa7c9091575836470a4434\": container with ID starting with fdf66999accbbe1ad4c116bb62c82a6764715ed865aa7c9091575836470a4434 not found: ID does not exist" Oct 30 08:55:15.491963 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.491932 5938 scope.go:122] "RemoveContainer" containerID="827272fdb9b7e83447fd6d35d96c4726bc1f60d78fd3d96158aacabc56c387f2" Oct 30 08:55:15.492149 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:55:15.492135 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"827272fdb9b7e83447fd6d35d96c4726bc1f60d78fd3d96158aacabc56c387f2\": container with ID starting with 827272fdb9b7e83447fd6d35d96c4726bc1f60d78fd3d96158aacabc56c387f2 not found: ID does not exist" containerID="827272fdb9b7e83447fd6d35d96c4726bc1f60d78fd3d96158aacabc56c387f2" Oct 30 08:55:15.492195 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.492154 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"827272fdb9b7e83447fd6d35d96c4726bc1f60d78fd3d96158aacabc56c387f2"} err="failed to get container status \"827272fdb9b7e83447fd6d35d96c4726bc1f60d78fd3d96158aacabc56c387f2\": rpc error: code = NotFound desc = could not find container \"827272fdb9b7e83447fd6d35d96c4726bc1f60d78fd3d96158aacabc56c387f2\": container with ID starting with 827272fdb9b7e83447fd6d35d96c4726bc1f60d78fd3d96158aacabc56c387f2 not found: ID does not exist" Oct 30 08:55:15.492195 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.492166 5938 scope.go:122] "RemoveContainer" containerID="428a41797386d649baf498c2ec11969043e66d787e4e260eb3d334cd7fc51391" Oct 30 08:55:15.492359 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:55:15.492345 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"428a41797386d649baf498c2ec11969043e66d787e4e260eb3d334cd7fc51391\": container with ID starting with 428a41797386d649baf498c2ec11969043e66d787e4e260eb3d334cd7fc51391 not found: ID does not exist" containerID="428a41797386d649baf498c2ec11969043e66d787e4e260eb3d334cd7fc51391" Oct 30 08:55:15.492398 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.492362 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428a41797386d649baf498c2ec11969043e66d787e4e260eb3d334cd7fc51391"} err="failed to get container status \"428a41797386d649baf498c2ec11969043e66d787e4e260eb3d334cd7fc51391\": rpc error: code = NotFound desc = could not find container \"428a41797386d649baf498c2ec11969043e66d787e4e260eb3d334cd7fc51391\": container with ID starting with 428a41797386d649baf498c2ec11969043e66d787e4e260eb3d334cd7fc51391 not found: ID does not exist" Oct 30 08:55:15.497198 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.497181 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 08:55:15.510860 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.510836 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 08:55:15.527080 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527053 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 08:55:15.527416 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527396 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="kube-rbac-proxy-web" Oct 30 08:55:15.527416 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527417 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="kube-rbac-proxy-web" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527437 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="kube-rbac-proxy-metric" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527444 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="kube-rbac-proxy-metric" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527456 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="kube-rbac-proxy" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527463 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="kube-rbac-proxy" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527480 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="init-config-reloader" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527486 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="init-config-reloader" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527512 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="prom-label-proxy" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527518 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="prom-label-proxy" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527530 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="config-reloader" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527536 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="config-reloader" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527550 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="alertmanager" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527558 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="alertmanager" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527632 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="kube-rbac-proxy-web" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527643 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="kube-rbac-proxy-metric" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527649 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="kube-rbac-proxy" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527657 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="init-config-reloader" Oct 30 08:55:15.527651 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527665 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="prom-label-proxy" Oct 30 08:55:15.528153 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527670 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="config-reloader" Oct 30 08:55:15.528153 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.527676 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" containerName="alertmanager" Oct 30 08:55:15.531162 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.531145 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.547436 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.547417 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 08:55:15.547518 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.547435 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 08:55:15.547518 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.547439 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 08:55:15.547639 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.547537 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 08:55:15.547639 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.547554 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 08:55:15.547639 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.547560 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 08:55:15.547852 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.547827 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-752kq\"" Oct 30 08:55:15.547959 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.547869 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 08:55:15.547959 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.547895 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 08:55:15.566286 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.562913 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 08:55:15.566286 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.563202 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 08:55:15.629997 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.629972 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dkhd\" (UniqueName: \"kubernetes.io/projected/4eea09ac-fa40-47ea-affb-0e54be762c78-kube-api-access-7dkhd\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.630163 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.630005 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4eea09ac-fa40-47ea-affb-0e54be762c78-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.630163 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.630023 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4eea09ac-fa40-47ea-affb-0e54be762c78-config-out\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.630163 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.630041 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4eea09ac-fa40-47ea-affb-0e54be762c78-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.630163 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.630153 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eea09ac-fa40-47ea-affb-0e54be762c78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.630328 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.630188 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.630328 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.630209 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.630328 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.630228 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eea09ac-fa40-47ea-affb-0e54be762c78-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.630434 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.630331 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.630434 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.630353 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.630434 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.630377 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.630434 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.630399 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-config-volume\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.630628 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.630499 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-web-config\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.730839 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.730784 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.730839 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.730811 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.730839 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.730834 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.731046 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.730853 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-config-volume\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.731046 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.730881 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-web-config\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.731046 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.730915 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dkhd\" (UniqueName: \"kubernetes.io/projected/4eea09ac-fa40-47ea-affb-0e54be762c78-kube-api-access-7dkhd\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.731179 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.731081 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4eea09ac-fa40-47ea-affb-0e54be762c78-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.731179 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.731111 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4eea09ac-fa40-47ea-affb-0e54be762c78-config-out\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.731179 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.731134 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4eea09ac-fa40-47ea-affb-0e54be762c78-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.731310 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.731240 5938 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 08:55:15.731310 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.731249 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eea09ac-fa40-47ea-affb-0e54be762c78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.731310 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.731294 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.731463 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.731325 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.731463 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.731359 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eea09ac-fa40-47ea-affb-0e54be762c78-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.731579 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.731552 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4eea09ac-fa40-47ea-affb-0e54be762c78-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.732313 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.732286 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eea09ac-fa40-47ea-affb-0e54be762c78-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.732706 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.732517 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eea09ac-fa40-47ea-affb-0e54be762c78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.734306 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.734276 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4eea09ac-fa40-47ea-affb-0e54be762c78-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.734447 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.734426 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.734529 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.734433 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4eea09ac-fa40-47ea-affb-0e54be762c78-config-out\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.734529 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.734516 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.734666 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.734516 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.734666 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.734614 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.734852 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.734835 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.735247 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.735228 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-web-config\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.736143 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.736126 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4eea09ac-fa40-47ea-affb-0e54be762c78-config-volume\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.750739 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.750718 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dkhd\" (UniqueName: \"kubernetes.io/projected/4eea09ac-fa40-47ea-affb-0e54be762c78-kube-api-access-7dkhd\") pod \"alertmanager-main-0\" (UID: \"4eea09ac-fa40-47ea-affb-0e54be762c78\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:15.806148 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:55:15.806120 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 08:55:15.857686 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:15.857669 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 08:55:16.190482 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:16.190453 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 08:55:16.193545 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:55:16.193520 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eea09ac_fa40_47ea_affb_0e54be762c78.slice/crio-0e54f5a59ba8ce07ee0b5ee24431b0bf09633b5dc60c7e60a20141946fd6719a WatchSource:0}: Error finding container 0e54f5a59ba8ce07ee0b5ee24431b0bf09633b5dc60c7e60a20141946fd6719a: Status 404 returned error can't find the container with id 0e54f5a59ba8ce07ee0b5ee24431b0bf09633b5dc60c7e60a20141946fd6719a Oct 30 08:55:16.255267 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:16.255234 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ac551a-4850-4c58-9fe8-6432e5d3301c" path="/var/lib/kubelet/pods/b0ac551a-4850-4c58-9fe8-6432e5d3301c/volumes" Oct 30 08:55:16.447620 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:16.447509 5938 generic.go:356] "Generic (PLEG): container finished" podID="4eea09ac-fa40-47ea-affb-0e54be762c78" containerID="b7f473a67eebe85b930f0bc28f27d38081011ccfe7c7c3aec18f2d1cf2dcd097" exitCode=0 Oct 30 08:55:16.447758 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:16.447611 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4eea09ac-fa40-47ea-affb-0e54be762c78","Type":"ContainerDied","Data":"b7f473a67eebe85b930f0bc28f27d38081011ccfe7c7c3aec18f2d1cf2dcd097"} Oct 30 08:55:16.447758 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:16.447681 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4eea09ac-fa40-47ea-affb-0e54be762c78","Type":"ContainerStarted","Data":"0e54f5a59ba8ce07ee0b5ee24431b0bf09633b5dc60c7e60a20141946fd6719a"} Oct 30 08:55:17.456911 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:17.456880 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4eea09ac-fa40-47ea-affb-0e54be762c78","Type":"ContainerStarted","Data":"4c0599497f59d4b6f0c642d3635e82bf4785e849bfc4dab7b1ffc783a1ce62b7"} Oct 30 08:55:17.457343 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:17.456941 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4eea09ac-fa40-47ea-affb-0e54be762c78","Type":"ContainerStarted","Data":"9cb6923f2c1cb790f5eb08103dc387db67059cb854989a4c96a3ef89577b286e"} Oct 30 08:55:17.457343 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:17.456967 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4eea09ac-fa40-47ea-affb-0e54be762c78","Type":"ContainerStarted","Data":"4aee00d9c22f1d7f83dcf14b64c8a2a051378ae876fc0fe970b19713ecd60ae4"} Oct 30 08:55:17.457343 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:17.456991 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4eea09ac-fa40-47ea-affb-0e54be762c78","Type":"ContainerStarted","Data":"53756bcdf312b2ec6d83944d136484f0016018f8a98fb36a7e3a757654218536"} Oct 30 08:55:17.457343 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:17.457013 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4eea09ac-fa40-47ea-affb-0e54be762c78","Type":"ContainerStarted","Data":"1bb408023aa4078dc15216416e7527ecaaae32ef05b6592d533c2945076a82e3"} Oct 30 08:55:17.457343 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:17.457037 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4eea09ac-fa40-47ea-affb-0e54be762c78","Type":"ContainerStarted","Data":"243cdcc53900bc6ccaf8eccfc4b7d96d3c41dce93a8a4fea46b3b1636a8fb674"} Oct 30 08:55:18.470809 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.470751 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.457811344 podStartE2EDuration="3.470736297s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.924953ms" isStatefulPod=false podCreationTimestamp="2026-10-30 08:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:55:17.546255738 +0000 UTC m=+35.408961560" watchObservedRunningTime="2026-10-30 08:55:18.470736297 +0000 UTC m=+36.333442109" Oct 30 08:55:18.471320 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.471300 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 08:55:18.471708 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.471681 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="prometheus" containerID="cri-o://62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c" gracePeriod=600 Oct 30 08:55:18.473712 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.473682 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="kube-rbac-proxy" containerID="cri-o://c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e" gracePeriod=600 Oct 30 08:55:18.473818 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.473718 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="thanos-sidecar" containerID="cri-o://afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119" gracePeriod=600 Oct 30 08:55:18.473818 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.473729 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="config-reloader" containerID="cri-o://0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661" gracePeriod=600 Oct 30 08:55:18.473938 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.473759 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="kube-rbac-proxy-web" containerID="cri-o://84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1" gracePeriod=600 Oct 30 08:55:18.474005 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.473777 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="kube-rbac-proxy-thanos" containerID="cri-o://78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a" gracePeriod=600 Oct 30 08:55:18.727887 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.727837 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:18.754385 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754360 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-1\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.754519 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754393 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-trusted-ca-bundle\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.754519 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754416 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-web-config\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-web-config\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.754519 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754445 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.754519 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754474 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-kube-rbac-proxy\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.754519 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754505 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-serving-certs-ca-bundle\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.754829 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754539 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.754829 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754578 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.754829 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754651 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-metrics-client-ca\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.754829 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754693 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-kube-api-access-tx5wx\" (UniqueName: \"kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-kube-api-access-tx5wx\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.754829 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754736 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-tls-assets\" (UniqueName: \"kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-tls-assets\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.754829 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754770 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-metrics-client-certs\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.754829 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754800 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-2\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.754829 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754800 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-1" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:18.755233 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754835 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-kubelet-serving-ca-bundle\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.755233 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754852 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-trusted-ca-bundle" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:18.755233 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754870 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config-out\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.755233 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.754931 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-serving-certs-ca-bundle" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:18.755612 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.755393 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-2" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:18.755612 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.755507 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-kubelet-serving-ca-bundle" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:18.756016 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.755991 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-0\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.756111 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.756048 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-grpc-tls\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.756111 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.756078 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-thanos-prometheus-http-client-file\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.756214 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.756115 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-db\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.756214 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.756150 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-tls\") pod \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\" (UID: \"9f15807d-cf9f-435a-84e6-febce2ef9ce4\") " Oct 30 08:55:18.756527 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.756487 5938 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.756627 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.756537 5938 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.756627 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.756555 5938 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.756627 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.756570 5938 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.756627 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.756619 5938 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.756843 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.756642 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-metrics-client-ca" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:18.758145 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.758117 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-db" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:55:18.759706 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.759683 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-0" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:18.759905 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.759883 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-kube-api-access-tx5wx" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "kube-api-access-tx5wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:55:18.760150 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.760131 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-thanos-sidecar-tls" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:18.760383 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.760367 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config-out" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:55:18.760383 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.760366 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-metrics-client-certs" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:18.760469 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.760451 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-kube-rbac-proxy" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:18.760552 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.760533 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-kube-rbac-proxy-web" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:18.760648 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.760625 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-grpc-tls" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:18.760715 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.760656 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:18.760770 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.760731 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-thanos-prometheus-http-client-file" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:18.760808 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.760780 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-tls-assets" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:55:18.761279 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.761264 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-tls" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:18.770311 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.770292 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-web-config" pod "9f15807d-cf9f-435a-84e6-febce2ef9ce4" (UID: "9f15807d-cf9f-435a-84e6-febce2ef9ce4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:18.858007 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.857975 5938 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-tls-assets\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.858007 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.858010 5938 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-metrics-client-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.858186 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.858025 5938 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config-out\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.858186 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.858039 5938 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.858186 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.858055 5938 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-grpc-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.858186 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.858071 5938 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.858186 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.858085 5938 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9f15807d-cf9f-435a-84e6-febce2ef9ce4-prometheus-k8s-db\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.858186 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.858098 5938 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.858186 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.858111 5938 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-web-config\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.858186 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.858124 5938 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.858186 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.858139 5938 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-kube-rbac-proxy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.858186 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.858150 5938 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-config\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.858186 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.858162 5938 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f15807d-cf9f-435a-84e6-febce2ef9ce4-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.858186 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.858177 5938 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f15807d-cf9f-435a-84e6-febce2ef9ce4-configmap-metrics-client-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:18.858186 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:18.858189 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx5wx\" (UniqueName: \"kubernetes.io/projected/9f15807d-cf9f-435a-84e6-febce2ef9ce4-kube-api-access-tx5wx\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:55:19.470534 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.470502 5938 generic.go:356] "Generic (PLEG): container finished" podID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerID="78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a" exitCode=0 Oct 30 08:55:19.470534 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.470526 5938 generic.go:356] "Generic (PLEG): container finished" podID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerID="c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e" exitCode=0 Oct 30 08:55:19.470534 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.470532 5938 generic.go:356] "Generic (PLEG): container finished" podID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerID="84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1" exitCode=0 Oct 30 08:55:19.470534 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.470537 5938 generic.go:356] "Generic (PLEG): container finished" podID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerID="afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119" exitCode=0 Oct 30 08:55:19.470534 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.470542 5938 generic.go:356] "Generic (PLEG): container finished" podID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerID="0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661" exitCode=0 Oct 30 08:55:19.470534 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.470547 5938 generic.go:356] "Generic (PLEG): container finished" podID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerID="62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c" exitCode=0 Oct 30 08:55:19.471401 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.470579 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f15807d-cf9f-435a-84e6-febce2ef9ce4","Type":"ContainerDied","Data":"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a"} Oct 30 08:55:19.471401 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.470630 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.471401 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.470659 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f15807d-cf9f-435a-84e6-febce2ef9ce4","Type":"ContainerDied","Data":"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e"} Oct 30 08:55:19.471401 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.470688 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f15807d-cf9f-435a-84e6-febce2ef9ce4","Type":"ContainerDied","Data":"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1"} Oct 30 08:55:19.471401 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.470713 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f15807d-cf9f-435a-84e6-febce2ef9ce4","Type":"ContainerDied","Data":"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119"} Oct 30 08:55:19.471401 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.470734 5938 scope.go:122] "RemoveContainer" containerID="78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a" Oct 30 08:55:19.471401 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.470737 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f15807d-cf9f-435a-84e6-febce2ef9ce4","Type":"ContainerDied","Data":"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661"} Oct 30 08:55:19.471401 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.470840 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f15807d-cf9f-435a-84e6-febce2ef9ce4","Type":"ContainerDied","Data":"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c"} Oct 30 08:55:19.471401 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.470866 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f15807d-cf9f-435a-84e6-febce2ef9ce4","Type":"ContainerDied","Data":"de8e1fa5a3d2bd96f1b430825b435f48cf9617c4ea7431323c387f9d1daddfe2"} Oct 30 08:55:19.479507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.479406 5938 scope.go:122] "RemoveContainer" containerID="c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e" Oct 30 08:55:19.496625 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.496605 5938 scope.go:122] "RemoveContainer" containerID="84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1" Oct 30 08:55:19.503326 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.503312 5938 scope.go:122] "RemoveContainer" containerID="afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119" Oct 30 08:55:19.509721 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.509703 5938 scope.go:122] "RemoveContainer" containerID="0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661" Oct 30 08:55:19.516086 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.516071 5938 scope.go:122] "RemoveContainer" containerID="62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c" Oct 30 08:55:19.522927 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.522914 5938 scope.go:122] "RemoveContainer" containerID="c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86" Oct 30 08:55:19.529323 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.529296 5938 scope.go:122] "RemoveContainer" containerID="78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a" Oct 30 08:55:19.529564 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:55:19.529547 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a\": container with ID starting with 78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a not found: ID does not exist" containerID="78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a" Oct 30 08:55:19.529643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.529572 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a"} err="failed to get container status \"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a\": rpc error: code = NotFound desc = could not find container \"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a\": container with ID starting with 78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a not found: ID does not exist" Oct 30 08:55:19.529643 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.529601 5938 scope.go:122] "RemoveContainer" containerID="c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e" Oct 30 08:55:19.529839 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:55:19.529824 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e\": container with ID starting with c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e not found: ID does not exist" containerID="c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e" Oct 30 08:55:19.529886 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.529841 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e"} err="failed to get container status \"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e\": rpc error: code = NotFound desc = could not find container \"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e\": container with ID starting with c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e not found: ID does not exist" Oct 30 08:55:19.529886 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.529853 5938 scope.go:122] "RemoveContainer" containerID="84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1" Oct 30 08:55:19.530075 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:55:19.530061 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1\": container with ID starting with 84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1 not found: ID does not exist" containerID="84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1" Oct 30 08:55:19.530116 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.530079 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1"} err="failed to get container status \"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1\": rpc error: code = NotFound desc = could not find container \"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1\": container with ID starting with 84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1 not found: ID does not exist" Oct 30 08:55:19.530116 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.530091 5938 scope.go:122] "RemoveContainer" containerID="afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119" Oct 30 08:55:19.530309 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:55:19.530293 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119\": container with ID starting with afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119 not found: ID does not exist" containerID="afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119" Oct 30 08:55:19.530354 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.530311 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119"} err="failed to get container status \"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119\": rpc error: code = NotFound desc = could not find container \"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119\": container with ID starting with afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119 not found: ID does not exist" Oct 30 08:55:19.530354 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.530322 5938 scope.go:122] "RemoveContainer" containerID="0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661" Oct 30 08:55:19.530530 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:55:19.530516 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661\": container with ID starting with 0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661 not found: ID does not exist" containerID="0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661" Oct 30 08:55:19.530564 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.530533 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661"} err="failed to get container status \"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661\": rpc error: code = NotFound desc = could not find container \"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661\": container with ID starting with 0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661 not found: ID does not exist" Oct 30 08:55:19.530564 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.530543 5938 scope.go:122] "RemoveContainer" containerID="62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c" Oct 30 08:55:19.530819 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:55:19.530804 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c\": container with ID starting with 62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c not found: ID does not exist" containerID="62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c" Oct 30 08:55:19.530876 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.530821 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c"} err="failed to get container status \"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c\": rpc error: code = NotFound desc = could not find container \"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c\": container with ID starting with 62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c not found: ID does not exist" Oct 30 08:55:19.530876 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.530832 5938 scope.go:122] "RemoveContainer" containerID="c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86" Oct 30 08:55:19.531039 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:55:19.531025 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86\": container with ID starting with c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86 not found: ID does not exist" containerID="c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86" Oct 30 08:55:19.531082 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.531042 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86"} err="failed to get container status \"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86\": rpc error: code = NotFound desc = could not find container \"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86\": container with ID starting with c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86 not found: ID does not exist" Oct 30 08:55:19.531082 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.531052 5938 scope.go:122] "RemoveContainer" containerID="78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a" Oct 30 08:55:19.531261 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.531246 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a"} err="failed to get container status \"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a\": rpc error: code = NotFound desc = could not find container \"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a\": container with ID starting with 78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a not found: ID does not exist" Oct 30 08:55:19.531305 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.531260 5938 scope.go:122] "RemoveContainer" containerID="c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e" Oct 30 08:55:19.531457 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.531442 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e"} err="failed to get container status \"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e\": rpc error: code = NotFound desc = could not find container \"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e\": container with ID starting with c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e not found: ID does not exist" Oct 30 08:55:19.531457 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.531456 5938 scope.go:122] "RemoveContainer" containerID="84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1" Oct 30 08:55:19.531692 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.531672 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1"} err="failed to get container status \"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1\": rpc error: code = NotFound desc = could not find container \"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1\": container with ID starting with 84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1 not found: ID does not exist" Oct 30 08:55:19.531752 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.531693 5938 scope.go:122] "RemoveContainer" containerID="afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119" Oct 30 08:55:19.531962 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.531947 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119"} err="failed to get container status \"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119\": rpc error: code = NotFound desc = could not find container \"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119\": container with ID starting with afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119 not found: ID does not exist" Oct 30 08:55:19.532004 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.531962 5938 scope.go:122] "RemoveContainer" containerID="0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661" Oct 30 08:55:19.532187 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.532167 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661"} err="failed to get container status \"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661\": rpc error: code = NotFound desc = could not find container \"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661\": container with ID starting with 0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661 not found: ID does not exist" Oct 30 08:55:19.532241 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.532188 5938 scope.go:122] "RemoveContainer" containerID="62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c" Oct 30 08:55:19.532411 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.532396 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c"} err="failed to get container status \"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c\": rpc error: code = NotFound desc = could not find container \"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c\": container with ID starting with 62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c not found: ID does not exist" Oct 30 08:55:19.532474 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.532412 5938 scope.go:122] "RemoveContainer" containerID="c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86" Oct 30 08:55:19.532661 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.532645 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86"} err="failed to get container status \"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86\": rpc error: code = NotFound desc = could not find container \"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86\": container with ID starting with c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86 not found: ID does not exist" Oct 30 08:55:19.532661 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.532661 5938 scope.go:122] "RemoveContainer" containerID="78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a" Oct 30 08:55:19.532872 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.532853 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a"} err="failed to get container status \"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a\": rpc error: code = NotFound desc = could not find container \"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a\": container with ID starting with 78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a not found: ID does not exist" Oct 30 08:55:19.532955 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.532875 5938 scope.go:122] "RemoveContainer" containerID="c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e" Oct 30 08:55:19.533090 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.533073 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e"} err="failed to get container status \"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e\": rpc error: code = NotFound desc = could not find container \"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e\": container with ID starting with c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e not found: ID does not exist" Oct 30 08:55:19.533136 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.533090 5938 scope.go:122] "RemoveContainer" containerID="84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1" Oct 30 08:55:19.533260 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.533243 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1"} err="failed to get container status \"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1\": rpc error: code = NotFound desc = could not find container \"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1\": container with ID starting with 84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1 not found: ID does not exist" Oct 30 08:55:19.533328 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.533262 5938 scope.go:122] "RemoveContainer" containerID="afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119" Oct 30 08:55:19.533527 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.533507 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119"} err="failed to get container status \"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119\": rpc error: code = NotFound desc = could not find container \"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119\": container with ID starting with afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119 not found: ID does not exist" Oct 30 08:55:19.533571 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.533528 5938 scope.go:122] "RemoveContainer" containerID="0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661" Oct 30 08:55:19.533765 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.533747 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661"} err="failed to get container status \"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661\": rpc error: code = NotFound desc = could not find container \"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661\": container with ID starting with 0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661 not found: ID does not exist" Oct 30 08:55:19.533834 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.533766 5938 scope.go:122] "RemoveContainer" containerID="62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c" Oct 30 08:55:19.534013 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.533996 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c"} err="failed to get container status \"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c\": rpc error: code = NotFound desc = could not find container \"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c\": container with ID starting with 62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c not found: ID does not exist" Oct 30 08:55:19.534056 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.534014 5938 scope.go:122] "RemoveContainer" containerID="c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86" Oct 30 08:55:19.534210 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.534193 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86"} err="failed to get container status \"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86\": rpc error: code = NotFound desc = could not find container \"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86\": container with ID starting with c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86 not found: ID does not exist" Oct 30 08:55:19.534277 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.534210 5938 scope.go:122] "RemoveContainer" containerID="78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a" Oct 30 08:55:19.534454 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.534438 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a"} err="failed to get container status \"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a\": rpc error: code = NotFound desc = could not find container \"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a\": container with ID starting with 78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a not found: ID does not exist" Oct 30 08:55:19.534499 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.534456 5938 scope.go:122] "RemoveContainer" containerID="c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e" Oct 30 08:55:19.534666 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.534650 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e"} err="failed to get container status \"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e\": rpc error: code = NotFound desc = could not find container \"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e\": container with ID starting with c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e not found: ID does not exist" Oct 30 08:55:19.534714 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.534666 5938 scope.go:122] "RemoveContainer" containerID="84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1" Oct 30 08:55:19.534860 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.534840 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1"} err="failed to get container status \"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1\": rpc error: code = NotFound desc = could not find container \"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1\": container with ID starting with 84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1 not found: ID does not exist" Oct 30 08:55:19.534860 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.534859 5938 scope.go:122] "RemoveContainer" containerID="afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119" Oct 30 08:55:19.535054 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.535036 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119"} err="failed to get container status \"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119\": rpc error: code = NotFound desc = could not find container \"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119\": container with ID starting with afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119 not found: ID does not exist" Oct 30 08:55:19.535097 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.535056 5938 scope.go:122] "RemoveContainer" containerID="0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661" Oct 30 08:55:19.535284 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.535266 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661"} err="failed to get container status \"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661\": rpc error: code = NotFound desc = could not find container \"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661\": container with ID starting with 0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661 not found: ID does not exist" Oct 30 08:55:19.535326 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.535284 5938 scope.go:122] "RemoveContainer" containerID="62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c" Oct 30 08:55:19.535505 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.535489 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c"} err="failed to get container status \"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c\": rpc error: code = NotFound desc = could not find container \"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c\": container with ID starting with 62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c not found: ID does not exist" Oct 30 08:55:19.535505 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.535504 5938 scope.go:122] "RemoveContainer" containerID="c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86" Oct 30 08:55:19.535741 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.535725 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86"} err="failed to get container status \"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86\": rpc error: code = NotFound desc = could not find container \"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86\": container with ID starting with c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86 not found: ID does not exist" Oct 30 08:55:19.535741 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.535740 5938 scope.go:122] "RemoveContainer" containerID="78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a" Oct 30 08:55:19.535981 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.535962 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a"} err="failed to get container status \"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a\": rpc error: code = NotFound desc = could not find container \"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a\": container with ID starting with 78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a not found: ID does not exist" Oct 30 08:55:19.536029 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.535981 5938 scope.go:122] "RemoveContainer" containerID="c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e" Oct 30 08:55:19.536212 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.536195 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e"} err="failed to get container status \"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e\": rpc error: code = NotFound desc = could not find container \"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e\": container with ID starting with c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e not found: ID does not exist" Oct 30 08:55:19.536212 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.536212 5938 scope.go:122] "RemoveContainer" containerID="84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1" Oct 30 08:55:19.536424 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.536405 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1"} err="failed to get container status \"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1\": rpc error: code = NotFound desc = could not find container \"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1\": container with ID starting with 84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1 not found: ID does not exist" Oct 30 08:55:19.536424 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.536423 5938 scope.go:122] "RemoveContainer" containerID="afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119" Oct 30 08:55:19.536636 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.536621 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119"} err="failed to get container status \"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119\": rpc error: code = NotFound desc = could not find container \"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119\": container with ID starting with afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119 not found: ID does not exist" Oct 30 08:55:19.536690 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.536636 5938 scope.go:122] "RemoveContainer" containerID="0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661" Oct 30 08:55:19.536846 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.536831 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661"} err="failed to get container status \"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661\": rpc error: code = NotFound desc = could not find container \"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661\": container with ID starting with 0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661 not found: ID does not exist" Oct 30 08:55:19.536846 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.536845 5938 scope.go:122] "RemoveContainer" containerID="62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c" Oct 30 08:55:19.537043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.537026 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c"} err="failed to get container status \"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c\": rpc error: code = NotFound desc = could not find container \"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c\": container with ID starting with 62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c not found: ID does not exist" Oct 30 08:55:19.537043 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.537042 5938 scope.go:122] "RemoveContainer" containerID="c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86" Oct 30 08:55:19.537290 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.537273 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86"} err="failed to get container status \"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86\": rpc error: code = NotFound desc = could not find container \"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86\": container with ID starting with c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86 not found: ID does not exist" Oct 30 08:55:19.537290 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.537289 5938 scope.go:122] "RemoveContainer" containerID="78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a" Oct 30 08:55:19.537623 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.537602 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a"} err="failed to get container status \"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a\": rpc error: code = NotFound desc = could not find container \"78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a\": container with ID starting with 78890479e6868d7fe68e65598824ead9443d583b74e59807039e1bb3b9b3a40a not found: ID does not exist" Oct 30 08:55:19.537623 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.537621 5938 scope.go:122] "RemoveContainer" containerID="c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e" Oct 30 08:55:19.537873 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.537856 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e"} err="failed to get container status \"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e\": rpc error: code = NotFound desc = could not find container \"c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e\": container with ID starting with c3989c0bbd55289afadff2426604c5d9040005c1ca00b2446edbcdd83d2f1c8e not found: ID does not exist" Oct 30 08:55:19.537873 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.537873 5938 scope.go:122] "RemoveContainer" containerID="84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1" Oct 30 08:55:19.538115 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.538096 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1"} err="failed to get container status \"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1\": rpc error: code = NotFound desc = could not find container \"84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1\": container with ID starting with 84a53024c5f7e6cb55b1fd947591fe47479a8e620c91463126fdf689080feca1 not found: ID does not exist" Oct 30 08:55:19.538176 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.538116 5938 scope.go:122] "RemoveContainer" containerID="afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119" Oct 30 08:55:19.538351 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.538330 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119"} err="failed to get container status \"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119\": rpc error: code = NotFound desc = could not find container \"afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119\": container with ID starting with afd79fda84a0ef3874560ac427931f6c552f42d40e3908d28ec80e9b70d36119 not found: ID does not exist" Oct 30 08:55:19.538400 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.538352 5938 scope.go:122] "RemoveContainer" containerID="0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661" Oct 30 08:55:19.538618 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.538602 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661"} err="failed to get container status \"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661\": rpc error: code = NotFound desc = could not find container \"0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661\": container with ID starting with 0b567d011d349f292c08a0ee8dedb25d1747106cd3d196cad9b367c0af1c7661 not found: ID does not exist" Oct 30 08:55:19.538618 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.538618 5938 scope.go:122] "RemoveContainer" containerID="62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c" Oct 30 08:55:19.538929 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.538905 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c"} err="failed to get container status \"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c\": rpc error: code = NotFound desc = could not find container \"62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c\": container with ID starting with 62fc25cad3b31396f50d1f41d8c4c75db3fda5717b9566bd7e3c6f0e8f4c457c not found: ID does not exist" Oct 30 08:55:19.539018 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.538930 5938 scope.go:122] "RemoveContainer" containerID="c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86" Oct 30 08:55:19.539209 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.539191 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86"} err="failed to get container status \"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86\": rpc error: code = NotFound desc = could not find container \"c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86\": container with ID starting with c6f2b84aa035443dae96a24df4d61368a5b3ff0d2da02c5970912adf98687e86 not found: ID does not exist" Oct 30 08:55:19.540550 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.540528 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 08:55:19.553608 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.553569 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 08:55:19.568442 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.568422 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.568861 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="kube-rbac-proxy" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.568880 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="kube-rbac-proxy" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.568898 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="prometheus" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.568905 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="prometheus" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.568919 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="config-reloader" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.568926 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="config-reloader" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.568939 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="init-config-reloader" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.568946 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="init-config-reloader" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.568956 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="kube-rbac-proxy-web" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.568962 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="kube-rbac-proxy-web" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.568983 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="thanos-sidecar" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.568990 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="thanos-sidecar" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.569003 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="kube-rbac-proxy-thanos" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.569009 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="kube-rbac-proxy-thanos" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.569079 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="thanos-sidecar" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.569088 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="kube-rbac-proxy-thanos" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.569100 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="kube-rbac-proxy" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.569108 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="prometheus" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.569116 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="config-reloader" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.569125 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="init-config-reloader" Oct 30 08:55:19.569206 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.569132 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" containerName="kube-rbac-proxy-web" Oct 30 08:55:19.574741 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.574724 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.592525 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.592499 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 08:55:19.592644 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.592533 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 08:55:19.592644 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.592542 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 08:55:19.592644 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.592626 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5ptj4\"" Oct 30 08:55:19.592829 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.592741 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 08:55:19.592930 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.592912 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 08:55:19.592991 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.592956 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 08:55:19.593042 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.593018 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 08:55:19.593042 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.593018 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 08:55:19.593042 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.593034 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 08:55:19.593042 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.593033 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 08:55:19.593220 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.593111 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 08:55:19.593220 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.593164 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3a6is943ja23r\"" Oct 30 08:55:19.593220 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.593172 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 08:55:19.593220 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.593204 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 08:55:19.607503 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.607291 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 08:55:19.607503 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.607473 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 08:55:19.608307 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.608201 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 08:55:19.665052 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665022 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665169 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665065 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665169 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665086 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665169 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665147 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665291 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665237 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665291 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665286 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665375 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665303 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665375 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665335 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-config\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665482 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665416 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d315846-1abf-45e2-80bd-07a0c1f2ac24-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665482 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665447 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d315846-1abf-45e2-80bd-07a0c1f2ac24-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665482 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665472 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtcx6\" (UniqueName: \"kubernetes.io/projected/0d315846-1abf-45e2-80bd-07a0c1f2ac24-kube-api-access-jtcx6\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665616 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665529 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665616 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665564 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d315846-1abf-45e2-80bd-07a0c1f2ac24-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665732 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665614 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665732 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665631 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665732 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665672 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665850 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665767 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665850 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665785 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665850 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665814 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.665850 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.665840 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767114 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767064 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767114 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767090 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-config\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767255 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767121 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d315846-1abf-45e2-80bd-07a0c1f2ac24-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767255 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767140 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d315846-1abf-45e2-80bd-07a0c1f2ac24-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767255 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767165 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtcx6\" (UniqueName: \"kubernetes.io/projected/0d315846-1abf-45e2-80bd-07a0c1f2ac24-kube-api-access-jtcx6\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767255 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767194 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767255 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767220 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d315846-1abf-45e2-80bd-07a0c1f2ac24-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767255 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767253 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767548 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767276 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767548 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767298 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767548 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767321 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767548 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767335 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767548 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767371 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767548 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767397 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767548 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767423 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767548 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767439 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767548 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767456 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767548 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767475 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.767548 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767514 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.768123 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767568 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.768123 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767714 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d315846-1abf-45e2-80bd-07a0c1f2ac24-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.768123 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.767837 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.768363 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.768120 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.768553 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.768525 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.768870 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.768846 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.770347 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.770316 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d315846-1abf-45e2-80bd-07a0c1f2ac24-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.771062 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.770643 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d315846-1abf-45e2-80bd-07a0c1f2ac24-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.771175 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.771152 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.771260 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.771242 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-config\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.771317 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.771293 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.771377 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.771335 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.771925 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.771850 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.772026 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.771957 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.772547 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.772519 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.773358 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.773335 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.773636 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.773375 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.773778 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.773761 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.773880 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.773863 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d315846-1abf-45e2-80bd-07a0c1f2ac24-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.774399 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.774379 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d315846-1abf-45e2-80bd-07a0c1f2ac24-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.787244 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.787225 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtcx6\" (UniqueName: \"kubernetes.io/projected/0d315846-1abf-45e2-80bd-07a0c1f2ac24-kube-api-access-jtcx6\") pod \"prometheus-k8s-0\" (UID: \"0d315846-1abf-45e2-80bd-07a0c1f2ac24\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:19.899885 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:19.899853 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:20.226564 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:20.226538 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 08:55:20.230062 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:55:20.230036 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d315846_1abf_45e2_80bd_07a0c1f2ac24.slice/crio-6fb7c1bcc2f062f671cac55c157f0012d8f3bb5ec5466f861f298f6255ab22a9 WatchSource:0}: Error finding container 6fb7c1bcc2f062f671cac55c157f0012d8f3bb5ec5466f861f298f6255ab22a9: Status 404 returned error can't find the container with id 6fb7c1bcc2f062f671cac55c157f0012d8f3bb5ec5466f861f298f6255ab22a9 Oct 30 08:55:20.254674 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:20.254653 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f15807d-cf9f-435a-84e6-febce2ef9ce4" path="/var/lib/kubelet/pods/9f15807d-cf9f-435a-84e6-febce2ef9ce4/volumes" Oct 30 08:55:20.475570 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:20.475543 5938 generic.go:356] "Generic (PLEG): container finished" podID="0d315846-1abf-45e2-80bd-07a0c1f2ac24" containerID="4e99379de23b10c3e93cbb7ca223780eb98835fcc1e498042df4ac2022cffd3f" exitCode=0 Oct 30 08:55:20.475941 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:20.475627 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d315846-1abf-45e2-80bd-07a0c1f2ac24","Type":"ContainerDied","Data":"4e99379de23b10c3e93cbb7ca223780eb98835fcc1e498042df4ac2022cffd3f"} Oct 30 08:55:20.475941 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:20.475699 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d315846-1abf-45e2-80bd-07a0c1f2ac24","Type":"ContainerStarted","Data":"6fb7c1bcc2f062f671cac55c157f0012d8f3bb5ec5466f861f298f6255ab22a9"} Oct 30 08:55:21.484104 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:21.484064 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d315846-1abf-45e2-80bd-07a0c1f2ac24","Type":"ContainerStarted","Data":"3d8c17fb7aa1d68eb1eba1101a6a4baf1c3b041db55e16e3d4b8d0370dc70224"} Oct 30 08:55:21.484467 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:21.484196 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d315846-1abf-45e2-80bd-07a0c1f2ac24","Type":"ContainerStarted","Data":"c0c689e8f8e8428dbda94d690520b0ebb2c736bbce3328124813fea52361b393"} Oct 30 08:55:21.484467 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:21.484239 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d315846-1abf-45e2-80bd-07a0c1f2ac24","Type":"ContainerStarted","Data":"8df581403f0d159bbc592f61eb72506d1b495d34ef2eed8d4fa1c1b70e041da2"} Oct 30 08:55:21.484467 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:21.484287 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d315846-1abf-45e2-80bd-07a0c1f2ac24","Type":"ContainerStarted","Data":"e79ec78c38d410b5c3d1f3bb36e4c16383b5a7b78661d1a28770e3c1054ca718"} Oct 30 08:55:21.484467 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:21.484327 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d315846-1abf-45e2-80bd-07a0c1f2ac24","Type":"ContainerStarted","Data":"aac50560e90896c57b8a47012bdf99189442facf2a3a8279ea5a84e946ce5aa2"} Oct 30 08:55:21.484467 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:21.484370 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d315846-1abf-45e2-80bd-07a0c1f2ac24","Type":"ContainerStarted","Data":"efb548921f77232ef0959a3681bb90d06a1d6dd710d5df70410486cf7b531cad"} Oct 30 08:55:21.547764 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:21.547721 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.528862867 podStartE2EDuration="2.547709826s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.846959ms" isStatefulPod=false podCreationTimestamp="2026-10-30 08:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:55:21.546957255 +0000 UTC m=+39.409663101" watchObservedRunningTime="2026-10-30 08:55:21.547709826 +0000 UTC m=+39.410415647" Oct 30 08:55:24.900679 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:55:24.900651 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:55:45.813027 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:55:45.812987 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 08:56:13.386582 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:56:13.386550 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:56:13.402785 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:56:13.402761 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:56:13.697960 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:56:13.697903 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 08:56:15.806548 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:56:15.806513 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 08:56:45.806914 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:56:45.806876 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 08:57:15.807078 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:57:15.807041 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 08:57:45.806770 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:57:45.806734 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 08:57:49.197399 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.197368 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb"] Oct 30 08:57:49.200776 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.200760 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" Oct 30 08:57:49.231114 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.231085 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-2rzfd\"" Oct 30 08:57:49.231232 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.231091 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 08:57:49.231232 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.231091 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 08:57:49.231784 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.231769 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb"] Oct 30 08:57:49.271856 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.271831 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb\" (UID: \"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" Oct 30 08:57:49.271953 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.271864 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb\" (UID: \"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" Oct 30 08:57:49.271994 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.271953 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5vgc\" (UniqueName: \"kubernetes.io/projected/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-kube-api-access-r5vgc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb\" (UID: \"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" Oct 30 08:57:49.372294 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.372270 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb\" (UID: \"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" Oct 30 08:57:49.372391 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.372301 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb\" (UID: \"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" Oct 30 08:57:49.372391 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.372345 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5vgc\" (UniqueName: \"kubernetes.io/projected/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-kube-api-access-r5vgc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb\" (UID: \"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" Oct 30 08:57:49.372654 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.372632 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb\" (UID: \"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" Oct 30 08:57:49.372727 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.372685 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb\" (UID: \"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" Oct 30 08:57:49.393287 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.393257 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5vgc\" (UniqueName: \"kubernetes.io/projected/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-kube-api-access-r5vgc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb\" (UID: \"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" Oct 30 08:57:49.520472 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.520425 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" Oct 30 08:57:49.812439 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.812375 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb"] Oct 30 08:57:49.815625 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:57:49.815571 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5dd299_cfe9_43fd_ba3c_ab26e7155bf4.slice/crio-712d7d803eee1eaf6eb71b0196b8252ced7f8e41297d682bb1b4d7f699032e8e WatchSource:0}: Error finding container 712d7d803eee1eaf6eb71b0196b8252ced7f8e41297d682bb1b4d7f699032e8e: Status 404 returned error can't find the container with id 712d7d803eee1eaf6eb71b0196b8252ced7f8e41297d682bb1b4d7f699032e8e Oct 30 08:57:49.817971 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:49.817955 5938 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 08:57:50.038141 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:50.038109 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" event={"ID":"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4","Type":"ContainerStarted","Data":"712d7d803eee1eaf6eb71b0196b8252ced7f8e41297d682bb1b4d7f699032e8e"} Oct 30 08:57:56.062847 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:56.062814 5938 generic.go:356] "Generic (PLEG): container finished" podID="8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4" containerID="5017f6bf5ad4745b860c74292621563f57200a38f3f451eaf79af04ddfa17f67" exitCode=0 Oct 30 08:57:56.063311 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:56.062862 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" event={"ID":"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4","Type":"ContainerDied","Data":"5017f6bf5ad4745b860c74292621563f57200a38f3f451eaf79af04ddfa17f67"} Oct 30 08:57:58.072777 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:58.072753 5938 generic.go:356] "Generic (PLEG): container finished" podID="8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4" containerID="0f86ebd6645859cf1014814f311841373ef8d2a47f2e9aa941af24eafdef0054" exitCode=0 Oct 30 08:57:58.073086 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:57:58.072824 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" event={"ID":"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4","Type":"ContainerDied","Data":"0f86ebd6645859cf1014814f311841373ef8d2a47f2e9aa941af24eafdef0054"} Oct 30 08:58:04.105870 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:04.105794 5938 generic.go:356] "Generic (PLEG): container finished" podID="8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4" containerID="0d71aba22e05bcf960b18d3654767e6ea8b4f7b868095b324f6bd8a4093c2ea6" exitCode=0 Oct 30 08:58:04.108459 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:04.105867 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" event={"ID":"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4","Type":"ContainerDied","Data":"0d71aba22e05bcf960b18d3654767e6ea8b4f7b868095b324f6bd8a4093c2ea6"} Oct 30 08:58:05.230389 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:05.230361 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" Oct 30 08:58:05.305098 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:05.305070 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-util\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-util\") pod \"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4\" (UID: \"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4\") " Oct 30 08:58:05.305282 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:05.305113 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-kube-api-access-r5vgc\" (UniqueName: \"kubernetes.io/projected/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-kube-api-access-r5vgc\") pod \"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4\" (UID: \"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4\") " Oct 30 08:58:05.305282 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:05.305137 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-bundle\") pod \"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4\" (UID: \"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4\") " Oct 30 08:58:05.305543 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:05.305515 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-bundle" pod "8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4" (UID: "8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:05.307442 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:05.307422 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-kube-api-access-r5vgc" pod "8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4" (UID: "8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4"). InnerVolumeSpecName "kube-api-access-r5vgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:58:05.309839 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:05.309818 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-util" pod "8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4" (UID: "8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:05.406554 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:05.406525 5938 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-util\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:58:05.406554 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:05.406553 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5vgc\" (UniqueName: \"kubernetes.io/projected/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-kube-api-access-r5vgc\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:58:05.406732 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:05.406568 5938 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4-bundle\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 08:58:06.114899 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:06.114863 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" event={"ID":"8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4","Type":"ContainerDied","Data":"712d7d803eee1eaf6eb71b0196b8252ced7f8e41297d682bb1b4d7f699032e8e"} Oct 30 08:58:06.114899 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:06.114893 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kttqb" Oct 30 08:58:06.115103 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:06.114917 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="712d7d803eee1eaf6eb71b0196b8252ced7f8e41297d682bb1b4d7f699032e8e" Oct 30 08:58:11.099568 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.099539 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-lwrbj"] Oct 30 08:58:11.102406 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.100578 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4" containerName="extract" Oct 30 08:58:11.102406 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.100632 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4" containerName="extract" Oct 30 08:58:11.102406 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.100686 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4" containerName="util" Oct 30 08:58:11.102406 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.100694 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4" containerName="util" Oct 30 08:58:11.102406 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.100712 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4" containerName="pull" Oct 30 08:58:11.102406 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.100720 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4" containerName="pull" Oct 30 08:58:11.102406 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.100898 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5dd299-cfe9-43fd-ba3c-ab26e7155bf4" containerName="extract" Oct 30 08:58:11.138233 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.138209 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-lwrbj"] Oct 30 08:58:11.138373 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.138288 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lwrbj" Oct 30 08:58:11.167984 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.167961 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:58:11.168247 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.168229 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 08:58:11.168336 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.168237 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-ncj5r\"" Oct 30 08:58:11.248204 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.248183 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwzds\" (UniqueName: \"kubernetes.io/projected/72a810d9-d6ed-467e-9bc1-b91074ccbe72-kube-api-access-fwzds\") pod \"jobset-operator-674cd59dc5-lwrbj\" (UID: \"72a810d9-d6ed-467e-9bc1-b91074ccbe72\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lwrbj" Oct 30 08:58:11.248286 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.248217 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72a810d9-d6ed-467e-9bc1-b91074ccbe72-tmp\") pod \"jobset-operator-674cd59dc5-lwrbj\" (UID: \"72a810d9-d6ed-467e-9bc1-b91074ccbe72\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lwrbj" Oct 30 08:58:11.348737 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.348707 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwzds\" (UniqueName: \"kubernetes.io/projected/72a810d9-d6ed-467e-9bc1-b91074ccbe72-kube-api-access-fwzds\") pod \"jobset-operator-674cd59dc5-lwrbj\" (UID: \"72a810d9-d6ed-467e-9bc1-b91074ccbe72\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lwrbj" Oct 30 08:58:11.348873 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.348749 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72a810d9-d6ed-467e-9bc1-b91074ccbe72-tmp\") pod \"jobset-operator-674cd59dc5-lwrbj\" (UID: \"72a810d9-d6ed-467e-9bc1-b91074ccbe72\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lwrbj" Oct 30 08:58:11.349067 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.349052 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72a810d9-d6ed-467e-9bc1-b91074ccbe72-tmp\") pod \"jobset-operator-674cd59dc5-lwrbj\" (UID: \"72a810d9-d6ed-467e-9bc1-b91074ccbe72\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lwrbj" Oct 30 08:58:11.369116 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.369093 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwzds\" (UniqueName: \"kubernetes.io/projected/72a810d9-d6ed-467e-9bc1-b91074ccbe72-kube-api-access-fwzds\") pod \"jobset-operator-674cd59dc5-lwrbj\" (UID: \"72a810d9-d6ed-467e-9bc1-b91074ccbe72\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lwrbj" Oct 30 08:58:11.474319 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.474298 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lwrbj" Oct 30 08:58:11.785918 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:11.785897 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-lwrbj"] Oct 30 08:58:11.788736 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:58:11.788709 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72a810d9_d6ed_467e_9bc1_b91074ccbe72.slice/crio-8ee1213726919ef5c3ed6ae1d62754b7bbfc2575d4a0d7c5b28c75167f0bd6cc WatchSource:0}: Error finding container 8ee1213726919ef5c3ed6ae1d62754b7bbfc2575d4a0d7c5b28c75167f0bd6cc: Status 404 returned error can't find the container with id 8ee1213726919ef5c3ed6ae1d62754b7bbfc2575d4a0d7c5b28c75167f0bd6cc Oct 30 08:58:12.140257 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:12.140223 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lwrbj" event={"ID":"72a810d9-d6ed-467e-9bc1-b91074ccbe72","Type":"ContainerStarted","Data":"8ee1213726919ef5c3ed6ae1d62754b7bbfc2575d4a0d7c5b28c75167f0bd6cc"} Oct 30 08:58:15.157932 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:15.157898 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lwrbj" event={"ID":"72a810d9-d6ed-467e-9bc1-b91074ccbe72","Type":"ContainerStarted","Data":"1a835a05206c86fc360b2a0913cc1e5dabcf8eeae1720ce64d73af870453b6b0"} Oct 30 08:58:15.199812 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:15.199685 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lwrbj" podStartSLOduration=1.843011181 podStartE2EDuration="4.199669103s" totalImagesPullingTime="2.356657922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:58:15.198537287 +0000 UTC m=+213.061243108" watchObservedRunningTime="2026-10-30 08:58:15.199669103 +0000 UTC m=+213.062374925" Oct 30 08:58:15.807037 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:58:15.807003 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 08:58:23.475522 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.475490 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229"] Oct 30 08:58:23.478846 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.478831 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.493420 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.493391 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229"] Oct 30 08:58:23.585496 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.585471 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 08:58:23.585753 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.585734 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 08:58:23.585808 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.585762 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 08:58:23.585854 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.585809 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 08:58:23.585992 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.585979 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 08:58:23.586055 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.586043 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-95xdf\"" Oct 30 08:58:23.643276 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.643255 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-manifests\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.643365 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.643282 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-webhook-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.643365 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.643304 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-tmp\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.643450 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.643418 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.643516 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.643497 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-metrics-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.643553 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.643529 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g7gq\" (UniqueName: \"kubernetes.io/projected/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-kube-api-access-9g7gq\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.744636 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.744566 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.744740 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.744638 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-metrics-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.744740 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.744665 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g7gq\" (UniqueName: \"kubernetes.io/projected/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-kube-api-access-9g7gq\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.744740 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.744698 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-manifests\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.744740 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.744728 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-webhook-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.744907 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.744759 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-tmp\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.745136 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.745116 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-manifests\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.747091 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.747065 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-tmp\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.747268 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.747246 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-metrics-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.747341 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.747269 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-webhook-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.747395 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.747349 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.767012 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.766993 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g7gq\" (UniqueName: \"kubernetes.io/projected/fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df-kube-api-access-9g7gq\") pod \"opendatahub-operator-controller-manager-69546f6877-9f229\" (UID: \"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:23.794054 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:23.794031 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:24.168825 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:24.168788 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229"] Oct 30 08:58:24.172999 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:58:24.172972 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc622ba8_c8b3_4ad9_aca2_dc7438a7f7df.slice/crio-e67ccfafae505af78d02d9f6bbea1053effb5b2ca1ad08502dec15c17d2a50cf WatchSource:0}: Error finding container e67ccfafae505af78d02d9f6bbea1053effb5b2ca1ad08502dec15c17d2a50cf: Status 404 returned error can't find the container with id e67ccfafae505af78d02d9f6bbea1053effb5b2ca1ad08502dec15c17d2a50cf Oct 30 08:58:24.193720 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:24.193691 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" event={"ID":"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df","Type":"ContainerStarted","Data":"e67ccfafae505af78d02d9f6bbea1053effb5b2ca1ad08502dec15c17d2a50cf"} Oct 30 08:58:27.207794 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:27.207723 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" event={"ID":"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df","Type":"ContainerStarted","Data":"b3e9b04b56f007b5208ccc59dbb98bc84e16150045c62221e80746a29b8dd649"} Oct 30 08:58:28.213830 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:28.213791 5938 generic.go:356] "Generic (PLEG): container finished" podID="fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df" containerID="b3e9b04b56f007b5208ccc59dbb98bc84e16150045c62221e80746a29b8dd649" exitCode=0 Oct 30 08:58:28.214287 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:28.213864 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" event={"ID":"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df","Type":"ContainerDied","Data":"b3e9b04b56f007b5208ccc59dbb98bc84e16150045c62221e80746a29b8dd649"} Oct 30 08:58:29.219763 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:29.219730 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" event={"ID":"fc622ba8-c8b3-4ad9-aca2-dc7438a7f7df","Type":"ContainerStarted","Data":"6503e07a5245dee29753e02a7c56993cd5828a1f276ed6b45ad20fc87c4fec02"} Oct 30 08:58:29.220129 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:29.219894 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:29.306703 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:29.306663 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" podStartSLOduration=2.732189801 podStartE2EDuration="6.306650978s" totalImagesPullingTime="3.085689219s" totalInitContainerRuntime="488.771958ms" isStatefulPod=false podCreationTimestamp="2026-10-30 08:58:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:58:29.29264994 +0000 UTC m=+227.155355760" watchObservedRunningTime="2026-10-30 08:58:29.306650978 +0000 UTC m=+227.169356795" Oct 30 08:58:37.818998 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:37.818963 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-nn4h9"] Oct 30 08:58:37.825022 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:37.825005 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nn4h9" Oct 30 08:58:37.843017 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:37.842968 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 08:58:37.843017 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:37.842984 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 08:58:37.843269 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:37.843075 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-tsp7m\"" Oct 30 08:58:37.843755 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:37.843727 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-nn4h9"] Oct 30 08:58:37.863475 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:37.863455 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znd7w\" (UniqueName: \"kubernetes.io/projected/cfccde0c-6f2b-4198-8ce5-b218d53f35d5-kube-api-access-znd7w\") pod \"cert-manager-cainjector-5c57bbd4fc-nn4h9\" (UID: \"cfccde0c-6f2b-4198-8ce5-b218d53f35d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nn4h9" Oct 30 08:58:37.863556 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:37.863518 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfccde0c-6f2b-4198-8ce5-b218d53f35d5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-nn4h9\" (UID: \"cfccde0c-6f2b-4198-8ce5-b218d53f35d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nn4h9" Oct 30 08:58:37.964000 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:37.963977 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfccde0c-6f2b-4198-8ce5-b218d53f35d5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-nn4h9\" (UID: \"cfccde0c-6f2b-4198-8ce5-b218d53f35d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nn4h9" Oct 30 08:58:37.964102 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:37.964019 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znd7w\" (UniqueName: \"kubernetes.io/projected/cfccde0c-6f2b-4198-8ce5-b218d53f35d5-kube-api-access-znd7w\") pod \"cert-manager-cainjector-5c57bbd4fc-nn4h9\" (UID: \"cfccde0c-6f2b-4198-8ce5-b218d53f35d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nn4h9" Oct 30 08:58:37.985053 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:37.985034 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znd7w\" (UniqueName: \"kubernetes.io/projected/cfccde0c-6f2b-4198-8ce5-b218d53f35d5-kube-api-access-znd7w\") pod \"cert-manager-cainjector-5c57bbd4fc-nn4h9\" (UID: \"cfccde0c-6f2b-4198-8ce5-b218d53f35d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nn4h9" Oct 30 08:58:37.985363 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:37.985346 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfccde0c-6f2b-4198-8ce5-b218d53f35d5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-nn4h9\" (UID: \"cfccde0c-6f2b-4198-8ce5-b218d53f35d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nn4h9" Oct 30 08:58:38.136545 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:38.136521 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nn4h9" Oct 30 08:58:38.432507 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:38.432441 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-nn4h9"] Oct 30 08:58:38.435543 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:58:38.435508 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfccde0c_6f2b_4198_8ce5_b218d53f35d5.slice/crio-e08b0063e3c08a36c7f257a482527a5ea0756fc55ac8bbace01ae98a4188cbe1 WatchSource:0}: Error finding container e08b0063e3c08a36c7f257a482527a5ea0756fc55ac8bbace01ae98a4188cbe1: Status 404 returned error can't find the container with id e08b0063e3c08a36c7f257a482527a5ea0756fc55ac8bbace01ae98a4188cbe1 Oct 30 08:58:39.262033 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:39.261990 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nn4h9" event={"ID":"cfccde0c-6f2b-4198-8ce5-b218d53f35d5","Type":"ContainerStarted","Data":"e08b0063e3c08a36c7f257a482527a5ea0756fc55ac8bbace01ae98a4188cbe1"} Oct 30 08:58:40.226931 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:40.226901 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9f229" Oct 30 08:58:42.275667 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:42.275638 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nn4h9" event={"ID":"cfccde0c-6f2b-4198-8ce5-b218d53f35d5","Type":"ContainerStarted","Data":"7ca8b38f7a111172e6346b1eaf97577ff82718653cc7030bc3a635e49b0361b1"} Oct 30 08:58:42.315273 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:42.315231 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nn4h9" podStartSLOduration=2.433131607 podStartE2EDuration="5.315218156s" totalImagesPullingTime="2.882086549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:58:42.315100151 +0000 UTC m=+240.177805972" watchObservedRunningTime="2026-10-30 08:58:42.315218156 +0000 UTC m=+240.177923976" Oct 30 08:58:45.624332 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:45.624299 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-hrwbd"] Oct 30 08:58:45.630572 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:45.630553 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hrwbd" Oct 30 08:58:45.646808 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:45.646778 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hrwbd"] Oct 30 08:58:45.659752 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:45.659730 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-8mr5k\"" Oct 30 08:58:45.733627 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:45.733575 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzdhs\" (UniqueName: \"kubernetes.io/projected/c2366273-6ba7-425b-8749-a8fa23cb498d-kube-api-access-fzdhs\") pod \"cert-manager-597b95cb84-hrwbd\" (UID: \"c2366273-6ba7-425b-8749-a8fa23cb498d\") " pod="cert-manager/cert-manager-597b95cb84-hrwbd" Oct 30 08:58:45.733730 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:45.733640 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2366273-6ba7-425b-8749-a8fa23cb498d-bound-sa-token\") pod \"cert-manager-597b95cb84-hrwbd\" (UID: \"c2366273-6ba7-425b-8749-a8fa23cb498d\") " pod="cert-manager/cert-manager-597b95cb84-hrwbd" Oct 30 08:58:45.806947 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:58:45.806918 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 08:58:45.834371 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:45.834348 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzdhs\" (UniqueName: \"kubernetes.io/projected/c2366273-6ba7-425b-8749-a8fa23cb498d-kube-api-access-fzdhs\") pod \"cert-manager-597b95cb84-hrwbd\" (UID: \"c2366273-6ba7-425b-8749-a8fa23cb498d\") " pod="cert-manager/cert-manager-597b95cb84-hrwbd" Oct 30 08:58:45.834456 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:45.834381 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2366273-6ba7-425b-8749-a8fa23cb498d-bound-sa-token\") pod \"cert-manager-597b95cb84-hrwbd\" (UID: \"c2366273-6ba7-425b-8749-a8fa23cb498d\") " pod="cert-manager/cert-manager-597b95cb84-hrwbd" Oct 30 08:58:45.855654 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:45.855634 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2366273-6ba7-425b-8749-a8fa23cb498d-bound-sa-token\") pod \"cert-manager-597b95cb84-hrwbd\" (UID: \"c2366273-6ba7-425b-8749-a8fa23cb498d\") " pod="cert-manager/cert-manager-597b95cb84-hrwbd" Oct 30 08:58:45.855796 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:45.855781 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzdhs\" (UniqueName: \"kubernetes.io/projected/c2366273-6ba7-425b-8749-a8fa23cb498d-kube-api-access-fzdhs\") pod \"cert-manager-597b95cb84-hrwbd\" (UID: \"c2366273-6ba7-425b-8749-a8fa23cb498d\") " pod="cert-manager/cert-manager-597b95cb84-hrwbd" Oct 30 08:58:45.942562 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:45.942513 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hrwbd" Oct 30 08:58:46.238973 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:46.238853 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hrwbd"] Oct 30 08:58:46.247689 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:58:46.247650 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2366273_6ba7_425b_8749_a8fa23cb498d.slice/crio-61809768fa4590252f5e7cc00b2c516c7f50419bf255f523b1f00c7a3772f63f WatchSource:0}: Error finding container 61809768fa4590252f5e7cc00b2c516c7f50419bf255f523b1f00c7a3772f63f: Status 404 returned error can't find the container with id 61809768fa4590252f5e7cc00b2c516c7f50419bf255f523b1f00c7a3772f63f Oct 30 08:58:46.291703 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:46.291673 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hrwbd" event={"ID":"c2366273-6ba7-425b-8749-a8fa23cb498d","Type":"ContainerStarted","Data":"61809768fa4590252f5e7cc00b2c516c7f50419bf255f523b1f00c7a3772f63f"} Oct 30 08:58:47.297976 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:47.297942 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hrwbd" event={"ID":"c2366273-6ba7-425b-8749-a8fa23cb498d","Type":"ContainerStarted","Data":"337953dd5cc9bacc8b013f25eb9d9a1f22a27a6196afde451a39f8ceda58b754"} Oct 30 08:58:47.340111 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:47.340070 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-hrwbd" podStartSLOduration=2.340057972 podStartE2EDuration="2.340057972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:58:47.339380243 +0000 UTC m=+245.202086065" watchObservedRunningTime="2026-10-30 08:58:47.340057972 +0000 UTC m=+245.202763783" Oct 30 08:58:58.081836 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.081797 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw"] Oct 30 08:58:58.085660 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.085637 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw" Oct 30 08:58:58.105735 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.105701 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 08:58:58.119240 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.119212 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 08:58:58.120789 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.120765 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw"] Oct 30 08:58:58.198934 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.198913 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 08:58:58.198934 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.198930 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 08:58:58.199116 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.199012 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-h299b\"" Oct 30 08:58:58.230624 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.230579 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twwfp\" (UniqueName: \"kubernetes.io/projected/12b2c3e7-3797-46e8-85df-93d84f58d580-kube-api-access-twwfp\") pod \"kube-auth-proxy-767d5cbf5-8tqvw\" (UID: \"12b2c3e7-3797-46e8-85df-93d84f58d580\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw" Oct 30 08:58:58.230747 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.230628 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12b2c3e7-3797-46e8-85df-93d84f58d580-tmp\") pod \"kube-auth-proxy-767d5cbf5-8tqvw\" (UID: \"12b2c3e7-3797-46e8-85df-93d84f58d580\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw" Oct 30 08:58:58.230747 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.230660 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b2c3e7-3797-46e8-85df-93d84f58d580-tls-certs\") pod \"kube-auth-proxy-767d5cbf5-8tqvw\" (UID: \"12b2c3e7-3797-46e8-85df-93d84f58d580\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw" Oct 30 08:58:58.331637 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.331607 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twwfp\" (UniqueName: \"kubernetes.io/projected/12b2c3e7-3797-46e8-85df-93d84f58d580-kube-api-access-twwfp\") pod \"kube-auth-proxy-767d5cbf5-8tqvw\" (UID: \"12b2c3e7-3797-46e8-85df-93d84f58d580\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw" Oct 30 08:58:58.331797 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.331659 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12b2c3e7-3797-46e8-85df-93d84f58d580-tmp\") pod \"kube-auth-proxy-767d5cbf5-8tqvw\" (UID: \"12b2c3e7-3797-46e8-85df-93d84f58d580\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw" Oct 30 08:58:58.331797 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.331710 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b2c3e7-3797-46e8-85df-93d84f58d580-tls-certs\") pod \"kube-auth-proxy-767d5cbf5-8tqvw\" (UID: \"12b2c3e7-3797-46e8-85df-93d84f58d580\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw" Oct 30 08:58:58.334212 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.334146 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12b2c3e7-3797-46e8-85df-93d84f58d580-tmp\") pod \"kube-auth-proxy-767d5cbf5-8tqvw\" (UID: \"12b2c3e7-3797-46e8-85df-93d84f58d580\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw" Oct 30 08:58:58.334449 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.334429 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b2c3e7-3797-46e8-85df-93d84f58d580-tls-certs\") pod \"kube-auth-proxy-767d5cbf5-8tqvw\" (UID: \"12b2c3e7-3797-46e8-85df-93d84f58d580\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw" Oct 30 08:58:58.353284 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.353254 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twwfp\" (UniqueName: \"kubernetes.io/projected/12b2c3e7-3797-46e8-85df-93d84f58d580-kube-api-access-twwfp\") pod \"kube-auth-proxy-767d5cbf5-8tqvw\" (UID: \"12b2c3e7-3797-46e8-85df-93d84f58d580\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw" Oct 30 08:58:58.397530 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.397504 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw" Oct 30 08:58:58.703913 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:58.703889 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw"] Oct 30 08:58:58.706528 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:58:58.706499 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b2c3e7_3797_46e8_85df_93d84f58d580.slice/crio-e4b193d441395e400eb0028899c57eb428ae1ce15ca55d70b713f69c6e3ab67f WatchSource:0}: Error finding container e4b193d441395e400eb0028899c57eb428ae1ce15ca55d70b713f69c6e3ab67f: Status 404 returned error can't find the container with id e4b193d441395e400eb0028899c57eb428ae1ce15ca55d70b713f69c6e3ab67f Oct 30 08:58:59.357910 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:58:59.357867 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw" event={"ID":"12b2c3e7-3797-46e8-85df-93d84f58d580","Type":"ContainerStarted","Data":"e4b193d441395e400eb0028899c57eb428ae1ce15ca55d70b713f69c6e3ab67f"} Oct 30 08:59:01.369236 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:01.369206 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw" event={"ID":"12b2c3e7-3797-46e8-85df-93d84f58d580","Type":"ContainerStarted","Data":"bb5254e91a40a3d58327bd11276ea5255e23776d854d20cb614169aee4046632"} Oct 30 08:59:06.798380 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:06.798318 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-767d5cbf5-8tqvw" podStartSLOduration=6.575315305 podStartE2EDuration="8.798303093s" totalImagesPullingTime="2.222987788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:59:01.410022081 +0000 UTC m=+259.272727902" watchObservedRunningTime="2026-10-30 08:59:06.798303093 +0000 UTC m=+264.661008913" Oct 30 08:59:06.798957 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:06.798936 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4"] Oct 30 08:59:06.802492 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:06.802477 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" Oct 30 08:59:06.836346 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:06.836325 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 08:59:06.836346 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:06.836332 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-z8ssh\"" Oct 30 08:59:06.836492 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:06.836364 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 08:59:06.836492 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:06.836451 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 08:59:06.837172 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:06.837152 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4"] Oct 30 08:59:06.909396 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:06.909367 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg5mr\" (UniqueName: \"kubernetes.io/projected/c237b30d-683c-4b9f-b801-b5818127bcb5-kube-api-access-wg5mr\") pod \"jobset-controller-manager-55bdb968bb-g6td4\" (UID: \"c237b30d-683c-4b9f-b801-b5818127bcb5\") " pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" Oct 30 08:59:06.909578 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:06.909460 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c237b30d-683c-4b9f-b801-b5818127bcb5-cert\") pod \"jobset-controller-manager-55bdb968bb-g6td4\" (UID: \"c237b30d-683c-4b9f-b801-b5818127bcb5\") " pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" Oct 30 08:59:06.909578 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:06.909535 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c237b30d-683c-4b9f-b801-b5818127bcb5-manager-config\") pod \"jobset-controller-manager-55bdb968bb-g6td4\" (UID: \"c237b30d-683c-4b9f-b801-b5818127bcb5\") " pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" Oct 30 08:59:06.909695 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:06.909645 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c237b30d-683c-4b9f-b801-b5818127bcb5-metrics-certs\") pod \"jobset-controller-manager-55bdb968bb-g6td4\" (UID: \"c237b30d-683c-4b9f-b801-b5818127bcb5\") " pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" Oct 30 08:59:07.011130 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:07.011097 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c237b30d-683c-4b9f-b801-b5818127bcb5-cert\") pod \"jobset-controller-manager-55bdb968bb-g6td4\" (UID: \"c237b30d-683c-4b9f-b801-b5818127bcb5\") " pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" Oct 30 08:59:07.011298 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:07.011172 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c237b30d-683c-4b9f-b801-b5818127bcb5-manager-config\") pod \"jobset-controller-manager-55bdb968bb-g6td4\" (UID: \"c237b30d-683c-4b9f-b801-b5818127bcb5\") " pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" Oct 30 08:59:07.011298 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:07.011213 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c237b30d-683c-4b9f-b801-b5818127bcb5-metrics-certs\") pod \"jobset-controller-manager-55bdb968bb-g6td4\" (UID: \"c237b30d-683c-4b9f-b801-b5818127bcb5\") " pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" Oct 30 08:59:07.011451 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:07.011320 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg5mr\" (UniqueName: \"kubernetes.io/projected/c237b30d-683c-4b9f-b801-b5818127bcb5-kube-api-access-wg5mr\") pod \"jobset-controller-manager-55bdb968bb-g6td4\" (UID: \"c237b30d-683c-4b9f-b801-b5818127bcb5\") " pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" Oct 30 08:59:07.012843 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:07.012814 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c237b30d-683c-4b9f-b801-b5818127bcb5-manager-config\") pod \"jobset-controller-manager-55bdb968bb-g6td4\" (UID: \"c237b30d-683c-4b9f-b801-b5818127bcb5\") " pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" Oct 30 08:59:07.016372 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:07.015028 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c237b30d-683c-4b9f-b801-b5818127bcb5-cert\") pod \"jobset-controller-manager-55bdb968bb-g6td4\" (UID: \"c237b30d-683c-4b9f-b801-b5818127bcb5\") " pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" Oct 30 08:59:07.017665 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:07.017080 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c237b30d-683c-4b9f-b801-b5818127bcb5-metrics-certs\") pod \"jobset-controller-manager-55bdb968bb-g6td4\" (UID: \"c237b30d-683c-4b9f-b801-b5818127bcb5\") " pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" Oct 30 08:59:07.032300 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:07.032277 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg5mr\" (UniqueName: \"kubernetes.io/projected/c237b30d-683c-4b9f-b801-b5818127bcb5-kube-api-access-wg5mr\") pod \"jobset-controller-manager-55bdb968bb-g6td4\" (UID: \"c237b30d-683c-4b9f-b801-b5818127bcb5\") " pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" Oct 30 08:59:07.119729 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:07.116769 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" Oct 30 08:59:07.438186 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:07.438106 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4"] Oct 30 08:59:07.440803 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:59:07.440774 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc237b30d_683c_4b9f_b801_b5818127bcb5.slice/crio-cdbe820425d1147a55a86e8b0cd293832d893da497775d3c4b18c4eb3ade7ea6 WatchSource:0}: Error finding container cdbe820425d1147a55a86e8b0cd293832d893da497775d3c4b18c4eb3ade7ea6: Status 404 returned error can't find the container with id cdbe820425d1147a55a86e8b0cd293832d893da497775d3c4b18c4eb3ade7ea6 Oct 30 08:59:08.400478 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:08.400440 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" event={"ID":"c237b30d-683c-4b9f-b801-b5818127bcb5","Type":"ContainerStarted","Data":"cdbe820425d1147a55a86e8b0cd293832d893da497775d3c4b18c4eb3ade7ea6"} Oct 30 08:59:10.410494 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:10.410464 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" event={"ID":"c237b30d-683c-4b9f-b801-b5818127bcb5","Type":"ContainerStarted","Data":"f5466ba127db5425ce1b4a181cdce483716dc4e90c79ab7d27e3f2be83c6c0e7"} Oct 30 08:59:10.410913 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:10.410575 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" Oct 30 08:59:10.464005 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:10.463952 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" podStartSLOduration=1.5846987380000002 podStartE2EDuration="4.46393345s" totalImagesPullingTime="2.879234712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:59:10.462642422 +0000 UTC m=+268.325348243" watchObservedRunningTime="2026-10-30 08:59:10.46393345 +0000 UTC m=+268.326639273" Oct 30 08:59:15.807502 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:59:15.807466 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 08:59:21.419789 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:21.419727 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-55bdb968bb-g6td4" Oct 30 08:59:45.807646 ip-10-0-135-148 kubenswrapper[5938]: E1030 08:59:45.807610 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 08:59:57.004151 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.004104 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh"] Oct 30 08:59:57.007756 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.007729 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" Oct 30 08:59:57.026566 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.026544 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-f2hrw\"" Oct 30 08:59:57.027846 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.027825 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh"] Oct 30 08:59:57.040567 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.040543 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 08:59:57.040696 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.040652 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 08:59:57.048656 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.048640 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 08:59:57.087343 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.087317 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-dkbbc"] Oct 30 08:59:57.090798 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.090780 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-dkbbc" Oct 30 08:59:57.102516 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.102494 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxjrk\" (UniqueName: \"kubernetes.io/projected/277b4b12-0e74-4c12-9d45-e39128be819f-kube-api-access-gxjrk\") pod \"llmisvc-controller-manager-869d5dcc5d-sh5jh\" (UID: \"277b4b12-0e74-4c12-9d45-e39128be819f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" Oct 30 08:59:57.102614 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.102544 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/277b4b12-0e74-4c12-9d45-e39128be819f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sh5jh\" (UID: \"277b4b12-0e74-4c12-9d45-e39128be819f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" Oct 30 08:59:57.112654 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.112631 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-dkbbc"] Oct 30 08:59:57.126774 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.126754 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 08:59:57.126774 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.126761 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-f7h8z\"" Oct 30 08:59:57.126895 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.126787 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 08:59:57.203501 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.203477 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/277b4b12-0e74-4c12-9d45-e39128be819f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sh5jh\" (UID: \"277b4b12-0e74-4c12-9d45-e39128be819f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" Oct 30 08:59:57.203662 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.203521 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e57e45d7-2bbb-433c-aa34-e6f23c81efe1-cert\") pod \"odh-model-controller-769875db56-dkbbc\" (UID: \"e57e45d7-2bbb-433c-aa34-e6f23c81efe1\") " pod="opendatahub/odh-model-controller-769875db56-dkbbc" Oct 30 08:59:57.203662 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.203649 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jxf7\" (UniqueName: \"kubernetes.io/projected/e57e45d7-2bbb-433c-aa34-e6f23c81efe1-kube-api-access-5jxf7\") pod \"odh-model-controller-769875db56-dkbbc\" (UID: \"e57e45d7-2bbb-433c-aa34-e6f23c81efe1\") " pod="opendatahub/odh-model-controller-769875db56-dkbbc" Oct 30 08:59:57.203768 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.203742 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e57e45d7-2bbb-433c-aa34-e6f23c81efe1-metrics-certs\") pod \"odh-model-controller-769875db56-dkbbc\" (UID: \"e57e45d7-2bbb-433c-aa34-e6f23c81efe1\") " pod="opendatahub/odh-model-controller-769875db56-dkbbc" Oct 30 08:59:57.203813 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.203767 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxjrk\" (UniqueName: \"kubernetes.io/projected/277b4b12-0e74-4c12-9d45-e39128be819f-kube-api-access-gxjrk\") pod \"llmisvc-controller-manager-869d5dcc5d-sh5jh\" (UID: \"277b4b12-0e74-4c12-9d45-e39128be819f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" Oct 30 08:59:57.206175 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.206150 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/277b4b12-0e74-4c12-9d45-e39128be819f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sh5jh\" (UID: \"277b4b12-0e74-4c12-9d45-e39128be819f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" Oct 30 08:59:57.223845 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.223818 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxjrk\" (UniqueName: \"kubernetes.io/projected/277b4b12-0e74-4c12-9d45-e39128be819f-kube-api-access-gxjrk\") pod \"llmisvc-controller-manager-869d5dcc5d-sh5jh\" (UID: \"277b4b12-0e74-4c12-9d45-e39128be819f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" Oct 30 08:59:57.305173 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.305113 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e57e45d7-2bbb-433c-aa34-e6f23c81efe1-cert\") pod \"odh-model-controller-769875db56-dkbbc\" (UID: \"e57e45d7-2bbb-433c-aa34-e6f23c81efe1\") " pod="opendatahub/odh-model-controller-769875db56-dkbbc" Oct 30 08:59:57.305173 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.305160 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jxf7\" (UniqueName: \"kubernetes.io/projected/e57e45d7-2bbb-433c-aa34-e6f23c81efe1-kube-api-access-5jxf7\") pod \"odh-model-controller-769875db56-dkbbc\" (UID: \"e57e45d7-2bbb-433c-aa34-e6f23c81efe1\") " pod="opendatahub/odh-model-controller-769875db56-dkbbc" Oct 30 08:59:57.305296 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.305192 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e57e45d7-2bbb-433c-aa34-e6f23c81efe1-metrics-certs\") pod \"odh-model-controller-769875db56-dkbbc\" (UID: \"e57e45d7-2bbb-433c-aa34-e6f23c81efe1\") " pod="opendatahub/odh-model-controller-769875db56-dkbbc" Oct 30 08:59:57.307436 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.307415 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e57e45d7-2bbb-433c-aa34-e6f23c81efe1-cert\") pod \"odh-model-controller-769875db56-dkbbc\" (UID: \"e57e45d7-2bbb-433c-aa34-e6f23c81efe1\") " pod="opendatahub/odh-model-controller-769875db56-dkbbc" Oct 30 08:59:57.307637 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.307615 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e57e45d7-2bbb-433c-aa34-e6f23c81efe1-metrics-certs\") pod \"odh-model-controller-769875db56-dkbbc\" (UID: \"e57e45d7-2bbb-433c-aa34-e6f23c81efe1\") " pod="opendatahub/odh-model-controller-769875db56-dkbbc" Oct 30 08:59:57.320414 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.320390 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" Oct 30 08:59:57.326893 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.326870 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jxf7\" (UniqueName: \"kubernetes.io/projected/e57e45d7-2bbb-433c-aa34-e6f23c81efe1-kube-api-access-5jxf7\") pod \"odh-model-controller-769875db56-dkbbc\" (UID: \"e57e45d7-2bbb-433c-aa34-e6f23c81efe1\") " pod="opendatahub/odh-model-controller-769875db56-dkbbc" Oct 30 08:59:57.403108 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.403075 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-dkbbc" Oct 30 08:59:57.622010 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.621956 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh"] Oct 30 08:59:57.624273 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:59:57.624249 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod277b4b12_0e74_4c12_9d45_e39128be819f.slice/crio-e137537126d38f9e6a93a9882bf89300722561ffba5d6e7c7396cfe5818e91b4 WatchSource:0}: Error finding container e137537126d38f9e6a93a9882bf89300722561ffba5d6e7c7396cfe5818e91b4: Status 404 returned error can't find the container with id e137537126d38f9e6a93a9882bf89300722561ffba5d6e7c7396cfe5818e91b4 Oct 30 08:59:57.704130 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:57.704105 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-dkbbc"] Oct 30 08:59:57.706457 ip-10-0-135-148 kubenswrapper[5938]: W1030 08:59:57.706432 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57e45d7_2bbb_433c_aa34_e6f23c81efe1.slice/crio-f160793964f255a35e5191f963300550f027b6a9f9fb540636710cba9c5eb8fb WatchSource:0}: Error finding container f160793964f255a35e5191f963300550f027b6a9f9fb540636710cba9c5eb8fb: Status 404 returned error can't find the container with id f160793964f255a35e5191f963300550f027b6a9f9fb540636710cba9c5eb8fb Oct 30 08:59:58.624176 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:58.624136 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-dkbbc" event={"ID":"e57e45d7-2bbb-433c-aa34-e6f23c81efe1","Type":"ContainerStarted","Data":"f160793964f255a35e5191f963300550f027b6a9f9fb540636710cba9c5eb8fb"} Oct 30 08:59:58.626712 ip-10-0-135-148 kubenswrapper[5938]: I1030 08:59:58.626680 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" event={"ID":"277b4b12-0e74-4c12-9d45-e39128be819f","Type":"ContainerStarted","Data":"e137537126d38f9e6a93a9882bf89300722561ffba5d6e7c7396cfe5818e91b4"} Oct 30 09:00:00.640280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:00:00.640250 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" event={"ID":"277b4b12-0e74-4c12-9d45-e39128be819f","Type":"ContainerStarted","Data":"07a3f5200ea650191aba0959b1de2e92de7631d29dee12a7355676526bea26c1"} Oct 30 09:00:00.640779 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:00:00.640355 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" Oct 30 09:00:00.641762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:00:00.641737 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-dkbbc" event={"ID":"e57e45d7-2bbb-433c-aa34-e6f23c81efe1","Type":"ContainerStarted","Data":"112c9b29374e36507e6eadea2a0e846567a6e1af639083f3e453cc6494a1d34b"} Oct 30 09:00:00.641905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:00:00.641790 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-dkbbc" Oct 30 09:00:00.682504 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:00:00.682464 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" podStartSLOduration=2.602689868 podStartE2EDuration="4.682451007s" totalImagesPullingTime="2.079761139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:59:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:00:00.682035434 +0000 UTC m=+318.544741254" watchObservedRunningTime="2026-10-30 09:00:00.682451007 +0000 UTC m=+318.545156828" Oct 30 09:00:00.724441 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:00:00.724392 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-dkbbc" podStartSLOduration=1.070185992 podStartE2EDuration="3.724378847s" totalImagesPullingTime="2.654192855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:00:00.723894699 +0000 UTC m=+318.586600518" watchObservedRunningTime="2026-10-30 09:00:00.724378847 +0000 UTC m=+318.587084669" Oct 30 09:00:11.654031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:00:11.653991 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-dkbbc" Oct 30 09:00:15.807487 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:00:15.807450 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:00:31.654209 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:00:31.654180 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" Oct 30 09:00:45.807482 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:00:45.807444 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:01:15.806951 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:01:15.806867 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:01:45.806648 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:01:45.806611 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:02:15.806973 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:02:15.806938 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:02:45.807769 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:02:45.807719 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:03:15.808149 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:03:15.808112 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:03:45.806680 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:03:45.806647 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:04:15.806984 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:04:15.806896 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:04:45.806719 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:04:45.806682 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:05:15.807343 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:05:15.807306 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:05:45.807142 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:05:45.807097 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:06:15.807378 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:06:15.807339 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:06:45.807352 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:06:45.807317 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:07:15.807418 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:07:15.807328 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:07:45.806573 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:07:45.806533 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:08:15.806921 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:08:15.806887 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:08:45.808306 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:08:45.808202 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:09:15.807804 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:09:15.807763 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:09:38.769421 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:09:38.769385 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-b7d79"] Oct 30 09:09:38.772911 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:09:38.772893 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-b7d79" Oct 30 09:09:38.788312 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:09:38.788290 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-b7d79"] Oct 30 09:09:38.800984 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:09:38.800963 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cqn5p\"" Oct 30 09:09:38.800984 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:09:38.800963 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 09:09:38.801122 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:09:38.801019 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 09:09:38.819027 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:09:38.818993 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvqkx\" (UniqueName: \"kubernetes.io/projected/65d7d14e-b6cf-442e-bc32-74340b9ae8cc-kube-api-access-fvqkx\") pod \"authorino-56c6c9ddd5-b7d79\" (UID: \"65d7d14e-b6cf-442e-bc32-74340b9ae8cc\") " pod="kuadrant-system/authorino-56c6c9ddd5-b7d79" Oct 30 09:09:38.919935 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:09:38.919908 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvqkx\" (UniqueName: \"kubernetes.io/projected/65d7d14e-b6cf-442e-bc32-74340b9ae8cc-kube-api-access-fvqkx\") pod \"authorino-56c6c9ddd5-b7d79\" (UID: \"65d7d14e-b6cf-442e-bc32-74340b9ae8cc\") " pod="kuadrant-system/authorino-56c6c9ddd5-b7d79" Oct 30 09:09:38.940010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:09:38.939990 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvqkx\" (UniqueName: \"kubernetes.io/projected/65d7d14e-b6cf-442e-bc32-74340b9ae8cc-kube-api-access-fvqkx\") pod \"authorino-56c6c9ddd5-b7d79\" (UID: \"65d7d14e-b6cf-442e-bc32-74340b9ae8cc\") " pod="kuadrant-system/authorino-56c6c9ddd5-b7d79" Oct 30 09:09:39.083973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:09:39.083907 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-b7d79" Oct 30 09:09:39.375284 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:09:39.375249 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-b7d79"] Oct 30 09:09:39.378119 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:09:39.378095 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65d7d14e_b6cf_442e_bc32_74340b9ae8cc.slice/crio-8ab9fc10893fb51dc5dd27ac4a02ea163acd1eb4957da77289c9b618708a8123 WatchSource:0}: Error finding container 8ab9fc10893fb51dc5dd27ac4a02ea163acd1eb4957da77289c9b618708a8123: Status 404 returned error can't find the container with id 8ab9fc10893fb51dc5dd27ac4a02ea163acd1eb4957da77289c9b618708a8123 Oct 30 09:09:39.379503 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:09:39.379487 5938 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:09:40.163566 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:09:40.163523 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-b7d79" event={"ID":"65d7d14e-b6cf-442e-bc32-74340b9ae8cc","Type":"ContainerStarted","Data":"8ab9fc10893fb51dc5dd27ac4a02ea163acd1eb4957da77289c9b618708a8123"} Oct 30 09:09:42.174989 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:09:42.174950 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-b7d79" event={"ID":"65d7d14e-b6cf-442e-bc32-74340b9ae8cc","Type":"ContainerStarted","Data":"ed404501bf5777f26e96b68209808c1c5d4ed5a8514609ddf2e2678424668b11"} Oct 30 09:09:42.213063 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:09:42.213017 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-b7d79" podStartSLOduration=1.908287213 podStartE2EDuration="4.21300313s" totalImagesPullingTime="2.304715917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:09:42.212177453 +0000 UTC m=+900.074883277" watchObservedRunningTime="2026-10-30 09:09:42.21300313 +0000 UTC m=+900.075708955" Oct 30 09:09:45.807461 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:09:45.807430 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:10:15.807791 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:10:15.807713 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:10:45.806805 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:10:45.806762 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:10:56.636137 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.636103 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5d696658ff-242p9"] Oct 30 09:10:56.639669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.639650 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" Oct 30 09:10:56.656254 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.656229 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:10:56.669899 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.669879 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:10:56.670068 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.670051 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hx9dk\"" Oct 30 09:10:56.670270 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.670249 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 09:10:56.671364 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.671346 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d696658ff-242p9"] Oct 30 09:10:56.683538 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.683521 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 09:10:56.732846 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.732823 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e16e34a-4935-43de-b1db-af983af6fc0b-metrics-certs\") pod \"maas-api-5d696658ff-242p9\" (UID: \"8e16e34a-4935-43de-b1db-af983af6fc0b\") " pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" Oct 30 09:10:56.732946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.732872 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e16e34a-4935-43de-b1db-af983af6fc0b-maas-api-tls\") pod \"maas-api-5d696658ff-242p9\" (UID: \"8e16e34a-4935-43de-b1db-af983af6fc0b\") " pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" Oct 30 09:10:56.732946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.732901 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e16e34a-4935-43de-b1db-af983af6fc0b-openshift-service-ca\") pod \"maas-api-5d696658ff-242p9\" (UID: \"8e16e34a-4935-43de-b1db-af983af6fc0b\") " pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" Oct 30 09:10:56.733081 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.733060 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zb4l\" (UniqueName: \"kubernetes.io/projected/8e16e34a-4935-43de-b1db-af983af6fc0b-kube-api-access-4zb4l\") pod \"maas-api-5d696658ff-242p9\" (UID: \"8e16e34a-4935-43de-b1db-af983af6fc0b\") " pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" Oct 30 09:10:56.833744 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.833718 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e16e34a-4935-43de-b1db-af983af6fc0b-maas-api-tls\") pod \"maas-api-5d696658ff-242p9\" (UID: \"8e16e34a-4935-43de-b1db-af983af6fc0b\") " pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" Oct 30 09:10:56.833853 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.833769 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e16e34a-4935-43de-b1db-af983af6fc0b-openshift-service-ca\") pod \"maas-api-5d696658ff-242p9\" (UID: \"8e16e34a-4935-43de-b1db-af983af6fc0b\") " pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" Oct 30 09:10:56.833918 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.833865 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zb4l\" (UniqueName: \"kubernetes.io/projected/8e16e34a-4935-43de-b1db-af983af6fc0b-kube-api-access-4zb4l\") pod \"maas-api-5d696658ff-242p9\" (UID: \"8e16e34a-4935-43de-b1db-af983af6fc0b\") " pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" Oct 30 09:10:56.833918 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.833898 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e16e34a-4935-43de-b1db-af983af6fc0b-metrics-certs\") pod \"maas-api-5d696658ff-242p9\" (UID: \"8e16e34a-4935-43de-b1db-af983af6fc0b\") " pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" Oct 30 09:10:56.834465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.834447 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e16e34a-4935-43de-b1db-af983af6fc0b-openshift-service-ca\") pod \"maas-api-5d696658ff-242p9\" (UID: \"8e16e34a-4935-43de-b1db-af983af6fc0b\") " pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" Oct 30 09:10:56.836849 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.836830 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e16e34a-4935-43de-b1db-af983af6fc0b-metrics-certs\") pod \"maas-api-5d696658ff-242p9\" (UID: \"8e16e34a-4935-43de-b1db-af983af6fc0b\") " pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" Oct 30 09:10:56.837203 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.837183 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-844c8bbcf4-46cp6"] Oct 30 09:10:56.837292 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.837207 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e16e34a-4935-43de-b1db-af983af6fc0b-maas-api-tls\") pod \"maas-api-5d696658ff-242p9\" (UID: \"8e16e34a-4935-43de-b1db-af983af6fc0b\") " pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" Oct 30 09:10:56.841388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.841368 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-844c8bbcf4-46cp6" Oct 30 09:10:56.854095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.854071 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zb4l\" (UniqueName: \"kubernetes.io/projected/8e16e34a-4935-43de-b1db-af983af6fc0b-kube-api-access-4zb4l\") pod \"maas-api-5d696658ff-242p9\" (UID: \"8e16e34a-4935-43de-b1db-af983af6fc0b\") " pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" Oct 30 09:10:56.858913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.858892 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-844c8bbcf4-46cp6"] Oct 30 09:10:56.905460 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.905402 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 09:10:56.935039 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.935013 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e-tls-cert\") pod \"authorino-844c8bbcf4-46cp6\" (UID: \"20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e\") " pod="kuadrant-system/authorino-844c8bbcf4-46cp6" Oct 30 09:10:56.935164 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.935145 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69nn7\" (UniqueName: \"kubernetes.io/projected/20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e-kube-api-access-69nn7\") pod \"authorino-844c8bbcf4-46cp6\" (UID: \"20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e\") " pod="kuadrant-system/authorino-844c8bbcf4-46cp6" Oct 30 09:10:56.952985 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:56.952962 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" Oct 30 09:10:57.035975 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:57.035952 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e-tls-cert\") pod \"authorino-844c8bbcf4-46cp6\" (UID: \"20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e\") " pod="kuadrant-system/authorino-844c8bbcf4-46cp6" Oct 30 09:10:57.036087 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:57.036055 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69nn7\" (UniqueName: \"kubernetes.io/projected/20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e-kube-api-access-69nn7\") pod \"authorino-844c8bbcf4-46cp6\" (UID: \"20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e\") " pod="kuadrant-system/authorino-844c8bbcf4-46cp6" Oct 30 09:10:57.038359 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:57.038337 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e-tls-cert\") pod \"authorino-844c8bbcf4-46cp6\" (UID: \"20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e\") " pod="kuadrant-system/authorino-844c8bbcf4-46cp6" Oct 30 09:10:57.055723 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:57.055700 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69nn7\" (UniqueName: \"kubernetes.io/projected/20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e-kube-api-access-69nn7\") pod \"authorino-844c8bbcf4-46cp6\" (UID: \"20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e\") " pod="kuadrant-system/authorino-844c8bbcf4-46cp6" Oct 30 09:10:57.171785 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:57.171731 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-844c8bbcf4-46cp6" Oct 30 09:10:57.245786 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:57.245760 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d696658ff-242p9"] Oct 30 09:10:57.248485 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:10:57.248453 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e16e34a_4935_43de_b1db_af983af6fc0b.slice/crio-d0cda6c4102210a54c1b91d6e06caabb0b081266c1a5613e7bfd54b6993e6b31 WatchSource:0}: Error finding container d0cda6c4102210a54c1b91d6e06caabb0b081266c1a5613e7bfd54b6993e6b31: Status 404 returned error can't find the container with id d0cda6c4102210a54c1b91d6e06caabb0b081266c1a5613e7bfd54b6993e6b31 Oct 30 09:10:57.462791 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:57.462688 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-844c8bbcf4-46cp6"] Oct 30 09:10:57.465151 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:10:57.465124 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20dbcd2f_a195_4b2b_bbd9_9dc79e3eeb1e.slice/crio-d4c54327b336f8186eaedc6fd53381e0e49e7d969eb933e932c387f9f887499a WatchSource:0}: Error finding container d4c54327b336f8186eaedc6fd53381e0e49e7d969eb933e932c387f9f887499a: Status 404 returned error can't find the container with id d4c54327b336f8186eaedc6fd53381e0e49e7d969eb933e932c387f9f887499a Oct 30 09:10:57.496462 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:57.496431 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-844c8bbcf4-46cp6" event={"ID":"20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e","Type":"ContainerStarted","Data":"d4c54327b336f8186eaedc6fd53381e0e49e7d969eb933e932c387f9f887499a"} Oct 30 09:10:57.497575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:57.497545 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" event={"ID":"8e16e34a-4935-43de-b1db-af983af6fc0b","Type":"ContainerStarted","Data":"d0cda6c4102210a54c1b91d6e06caabb0b081266c1a5613e7bfd54b6993e6b31"} Oct 30 09:10:58.532098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:58.531630 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-844c8bbcf4-46cp6" event={"ID":"20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e","Type":"ContainerStarted","Data":"54021f650cd7871512988d88245c2a2240abb6d758be895a0f235e363e0965dc"} Oct 30 09:10:58.579545 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:58.579489 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-844c8bbcf4-46cp6" podStartSLOduration=1.891917064 podStartE2EDuration="2.579472705s" totalImagesPullingTime="687.555641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:10:58.578902249 +0000 UTC m=+976.441608070" watchObservedRunningTime="2026-10-30 09:10:58.579472705 +0000 UTC m=+976.442178527" Oct 30 09:10:58.605210 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:58.605174 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-b7d79"] Oct 30 09:10:58.605423 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:58.605397 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-b7d79" podUID="65d7d14e-b6cf-442e-bc32-74340b9ae8cc" containerName="authorino" containerID="cri-o://ed404501bf5777f26e96b68209808c1c5d4ed5a8514609ddf2e2678424668b11" gracePeriod=30 Oct 30 09:10:58.908235 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:58.908211 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-b7d79" Oct 30 09:10:58.958818 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:58.958785 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d7d14e-b6cf-442e-bc32-74340b9ae8cc-kube-api-access-fvqkx\" (UniqueName: \"kubernetes.io/projected/65d7d14e-b6cf-442e-bc32-74340b9ae8cc-kube-api-access-fvqkx\") pod \"65d7d14e-b6cf-442e-bc32-74340b9ae8cc\" (UID: \"65d7d14e-b6cf-442e-bc32-74340b9ae8cc\") " Oct 30 09:10:58.961790 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:58.961763 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d7d14e-b6cf-442e-bc32-74340b9ae8cc-kube-api-access-fvqkx" pod "65d7d14e-b6cf-442e-bc32-74340b9ae8cc" (UID: "65d7d14e-b6cf-442e-bc32-74340b9ae8cc"). InnerVolumeSpecName "kube-api-access-fvqkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:10:59.060289 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.060213 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvqkx\" (UniqueName: \"kubernetes.io/projected/65d7d14e-b6cf-442e-bc32-74340b9ae8cc-kube-api-access-fvqkx\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:10:59.539427 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.539391 5938 generic.go:356] "Generic (PLEG): container finished" podID="65d7d14e-b6cf-442e-bc32-74340b9ae8cc" containerID="ed404501bf5777f26e96b68209808c1c5d4ed5a8514609ddf2e2678424668b11" exitCode=0 Oct 30 09:10:59.539926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.539451 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-b7d79" Oct 30 09:10:59.539926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.539497 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-b7d79" event={"ID":"65d7d14e-b6cf-442e-bc32-74340b9ae8cc","Type":"ContainerDied","Data":"ed404501bf5777f26e96b68209808c1c5d4ed5a8514609ddf2e2678424668b11"} Oct 30 09:10:59.539926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.539562 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-b7d79" event={"ID":"65d7d14e-b6cf-442e-bc32-74340b9ae8cc","Type":"ContainerDied","Data":"8ab9fc10893fb51dc5dd27ac4a02ea163acd1eb4957da77289c9b618708a8123"} Oct 30 09:10:59.539926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.539608 5938 scope.go:122] "RemoveContainer" containerID="ed404501bf5777f26e96b68209808c1c5d4ed5a8514609ddf2e2678424668b11" Oct 30 09:10:59.549791 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.549774 5938 scope.go:122] "RemoveContainer" containerID="ed404501bf5777f26e96b68209808c1c5d4ed5a8514609ddf2e2678424668b11" Oct 30 09:10:59.550114 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:10:59.550081 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed404501bf5777f26e96b68209808c1c5d4ed5a8514609ddf2e2678424668b11\": container with ID starting with ed404501bf5777f26e96b68209808c1c5d4ed5a8514609ddf2e2678424668b11 not found: ID does not exist" containerID="ed404501bf5777f26e96b68209808c1c5d4ed5a8514609ddf2e2678424668b11" Oct 30 09:10:59.550217 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.550112 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed404501bf5777f26e96b68209808c1c5d4ed5a8514609ddf2e2678424668b11"} err="failed to get container status \"ed404501bf5777f26e96b68209808c1c5d4ed5a8514609ddf2e2678424668b11\": rpc error: code = NotFound desc = could not find container \"ed404501bf5777f26e96b68209808c1c5d4ed5a8514609ddf2e2678424668b11\": container with ID starting with ed404501bf5777f26e96b68209808c1c5d4ed5a8514609ddf2e2678424668b11 not found: ID does not exist" Oct 30 09:10:59.592143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.592119 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-b7d79"] Oct 30 09:10:59.600269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.600250 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-b7d79"] Oct 30 09:10:59.607172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.607152 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-f64pg"] Oct 30 09:10:59.607575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.607563 5938 cpu_manager.go:414] "containerMap: removing container" podUID="65d7d14e-b6cf-442e-bc32-74340b9ae8cc" containerName="authorino" Oct 30 09:10:59.607644 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.607577 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d7d14e-b6cf-442e-bc32-74340b9ae8cc" containerName="authorino" Oct 30 09:10:59.607708 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.607697 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d7d14e-b6cf-442e-bc32-74340b9ae8cc" containerName="authorino" Oct 30 09:10:59.610813 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.610799 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-f64pg" Oct 30 09:10:59.630633 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.630606 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-f64pg"] Oct 30 09:10:59.666527 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.666504 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1-cert\") pod \"llmisvc-controller-manager-6466c554d8-f64pg\" (UID: \"f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-f64pg" Oct 30 09:10:59.666658 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.666640 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmq4d\" (UniqueName: \"kubernetes.io/projected/f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1-kube-api-access-cmq4d\") pod \"llmisvc-controller-manager-6466c554d8-f64pg\" (UID: \"f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-f64pg" Oct 30 09:10:59.767595 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.767568 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmq4d\" (UniqueName: \"kubernetes.io/projected/f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1-kube-api-access-cmq4d\") pod \"llmisvc-controller-manager-6466c554d8-f64pg\" (UID: \"f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-f64pg" Oct 30 09:10:59.767717 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.767637 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1-cert\") pod \"llmisvc-controller-manager-6466c554d8-f64pg\" (UID: \"f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-f64pg" Oct 30 09:10:59.769967 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.769948 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1-cert\") pod \"llmisvc-controller-manager-6466c554d8-f64pg\" (UID: \"f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-f64pg" Oct 30 09:10:59.787541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.787517 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmq4d\" (UniqueName: \"kubernetes.io/projected/f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1-kube-api-access-cmq4d\") pod \"llmisvc-controller-manager-6466c554d8-f64pg\" (UID: \"f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-f64pg" Oct 30 09:10:59.923764 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:10:59.923745 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-f64pg" Oct 30 09:11:00.218365 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:00.218339 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-f64pg"] Oct 30 09:11:00.220755 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:11:00.220729 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fb5ced_7f2c_4dfb_91d4_8fef493cf1d1.slice/crio-8fe549ee9ba51f5c54f84ade55c9a77018c3fd009e3a27eccb64a858eebc2383 WatchSource:0}: Error finding container 8fe549ee9ba51f5c54f84ade55c9a77018c3fd009e3a27eccb64a858eebc2383: Status 404 returned error can't find the container with id 8fe549ee9ba51f5c54f84ade55c9a77018c3fd009e3a27eccb64a858eebc2383 Oct 30 09:11:00.255310 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:00.255278 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d7d14e-b6cf-442e-bc32-74340b9ae8cc" path="/var/lib/kubelet/pods/65d7d14e-b6cf-442e-bc32-74340b9ae8cc/volumes" Oct 30 09:11:00.545499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:00.545431 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" event={"ID":"8e16e34a-4935-43de-b1db-af983af6fc0b","Type":"ContainerStarted","Data":"aae0212f9ee7a80c5ca719b375a3f9b70bb7719fd268558bd75d50a51f961f8b"} Oct 30 09:11:00.545917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:00.545565 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" Oct 30 09:11:00.547633 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:00.547611 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-f64pg" event={"ID":"f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1","Type":"ContainerStarted","Data":"8fe549ee9ba51f5c54f84ade55c9a77018c3fd009e3a27eccb64a858eebc2383"} Oct 30 09:11:00.589833 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:00.589789 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" podStartSLOduration=2.180317856 podStartE2EDuration="4.589773171s" totalImagesPullingTime="2.409455315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:11:00.589446691 +0000 UTC m=+978.452152517" watchObservedRunningTime="2026-10-30 09:11:00.589773171 +0000 UTC m=+978.452478994" Oct 30 09:11:01.554417 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:01.554384 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-f64pg" event={"ID":"f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1","Type":"ContainerStarted","Data":"28a0184f292715f68232e13c11b83072c3383744a02bc2a78c0b891a927237ed"} Oct 30 09:11:01.554918 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:01.554448 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-f64pg" Oct 30 09:11:06.560117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:06.560088 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5d696658ff-242p9" Oct 30 09:11:06.602155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:06.602108 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-f64pg" podStartSLOduration=6.86246897 podStartE2EDuration="7.602093887s" totalImagesPullingTime="739.624917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:11:01.60736825 +0000 UTC m=+979.470074070" watchObservedRunningTime="2026-10-30 09:11:06.602093887 +0000 UTC m=+984.464799725" Oct 30 09:11:15.807505 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:11:15.807469 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:11:32.561364 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:32.561335 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-f64pg" Oct 30 09:11:32.628161 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:32.628131 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh"] Oct 30 09:11:32.628632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:32.628563 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" podUID="277b4b12-0e74-4c12-9d45-e39128be819f" containerName="manager" containerID="cri-o://07a3f5200ea650191aba0959b1de2e92de7631d29dee12a7355676526bea26c1" gracePeriod=30 Oct 30 09:11:32.873437 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:32.873414 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" Oct 30 09:11:33.042816 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:33.042786 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/277b4b12-0e74-4c12-9d45-e39128be819f-cert\" (UniqueName: \"kubernetes.io/secret/277b4b12-0e74-4c12-9d45-e39128be819f-cert\") pod \"277b4b12-0e74-4c12-9d45-e39128be819f\" (UID: \"277b4b12-0e74-4c12-9d45-e39128be819f\") " Oct 30 09:11:33.043042 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:33.042862 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277b4b12-0e74-4c12-9d45-e39128be819f-kube-api-access-gxjrk\" (UniqueName: \"kubernetes.io/projected/277b4b12-0e74-4c12-9d45-e39128be819f-kube-api-access-gxjrk\") pod \"277b4b12-0e74-4c12-9d45-e39128be819f\" (UID: \"277b4b12-0e74-4c12-9d45-e39128be819f\") " Oct 30 09:11:33.045020 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:33.044992 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/277b4b12-0e74-4c12-9d45-e39128be819f-cert" pod "277b4b12-0e74-4c12-9d45-e39128be819f" (UID: "277b4b12-0e74-4c12-9d45-e39128be819f"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:11:33.045128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:33.045026 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277b4b12-0e74-4c12-9d45-e39128be819f-kube-api-access-gxjrk" pod "277b4b12-0e74-4c12-9d45-e39128be819f" (UID: "277b4b12-0e74-4c12-9d45-e39128be819f"). InnerVolumeSpecName "kube-api-access-gxjrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:11:33.143579 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:33.143555 5938 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/277b4b12-0e74-4c12-9d45-e39128be819f-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:11:33.143723 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:33.143579 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxjrk\" (UniqueName: \"kubernetes.io/projected/277b4b12-0e74-4c12-9d45-e39128be819f-kube-api-access-gxjrk\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:11:33.702271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:33.702240 5938 generic.go:356] "Generic (PLEG): container finished" podID="277b4b12-0e74-4c12-9d45-e39128be819f" containerID="07a3f5200ea650191aba0959b1de2e92de7631d29dee12a7355676526bea26c1" exitCode=0 Oct 30 09:11:33.702709 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:33.702286 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" Oct 30 09:11:33.702709 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:33.702317 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" event={"ID":"277b4b12-0e74-4c12-9d45-e39128be819f","Type":"ContainerDied","Data":"07a3f5200ea650191aba0959b1de2e92de7631d29dee12a7355676526bea26c1"} Oct 30 09:11:33.702709 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:33.702368 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh" event={"ID":"277b4b12-0e74-4c12-9d45-e39128be819f","Type":"ContainerDied","Data":"e137537126d38f9e6a93a9882bf89300722561ffba5d6e7c7396cfe5818e91b4"} Oct 30 09:11:33.702709 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:33.702388 5938 scope.go:122] "RemoveContainer" containerID="07a3f5200ea650191aba0959b1de2e92de7631d29dee12a7355676526bea26c1" Oct 30 09:11:33.712150 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:33.712130 5938 scope.go:122] "RemoveContainer" containerID="07a3f5200ea650191aba0959b1de2e92de7631d29dee12a7355676526bea26c1" Oct 30 09:11:33.712457 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:11:33.712435 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a3f5200ea650191aba0959b1de2e92de7631d29dee12a7355676526bea26c1\": container with ID starting with 07a3f5200ea650191aba0959b1de2e92de7631d29dee12a7355676526bea26c1 not found: ID does not exist" containerID="07a3f5200ea650191aba0959b1de2e92de7631d29dee12a7355676526bea26c1" Oct 30 09:11:33.712503 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:33.712468 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a3f5200ea650191aba0959b1de2e92de7631d29dee12a7355676526bea26c1"} err="failed to get container status \"07a3f5200ea650191aba0959b1de2e92de7631d29dee12a7355676526bea26c1\": rpc error: code = NotFound desc = could not find container \"07a3f5200ea650191aba0959b1de2e92de7631d29dee12a7355676526bea26c1\": container with ID starting with 07a3f5200ea650191aba0959b1de2e92de7631d29dee12a7355676526bea26c1 not found: ID does not exist" Oct 30 09:11:33.757032 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:33.757005 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh"] Oct 30 09:11:33.759893 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:33.759872 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sh5jh"] Oct 30 09:11:34.261555 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:11:34.261518 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277b4b12-0e74-4c12-9d45-e39128be819f" path="/var/lib/kubelet/pods/277b4b12-0e74-4c12-9d45-e39128be819f/volumes" Oct 30 09:11:45.807753 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:11:45.807709 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:12:15.843237 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:12:15.843163 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:12:34.884785 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:34.884753 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6"] Oct 30 09:12:34.885283 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:34.885257 5938 cpu_manager.go:414] "containerMap: removing container" podUID="277b4b12-0e74-4c12-9d45-e39128be819f" containerName="manager" Oct 30 09:12:34.885283 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:34.885276 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277b4b12-0e74-4c12-9d45-e39128be819f" containerName="manager" Oct 30 09:12:34.885459 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:34.885371 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="277b4b12-0e74-4c12-9d45-e39128be819f" containerName="manager" Oct 30 09:12:34.888395 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:34.888377 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:34.907245 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:34.907225 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 09:12:34.931955 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:34.931921 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6"] Oct 30 09:12:34.997853 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:34.997830 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-j7g5p\"" Oct 30 09:12:35.078853 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.078822 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw59s\" (UniqueName: \"kubernetes.io/projected/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-kube-api-access-lw59s\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.079001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.078865 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.079001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.078887 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.079001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.078941 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.079001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.078987 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.079134 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.079051 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.079208 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.079147 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.079208 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.079200 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.079269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.079219 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.179979 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.179899 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.179979 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.179932 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.179979 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.179951 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw59s\" (UniqueName: \"kubernetes.io/projected/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-kube-api-access-lw59s\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.180238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.180077 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.180238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.180125 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.180238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.180150 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.180238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.180178 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.180238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.180203 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.180484 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.180301 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.180484 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.180402 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.180617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.180578 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.180796 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.180576 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.180905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.180856 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.181202 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.181184 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.182370 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.182349 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.182999 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.182980 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.203874 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.203850 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.204381 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.204366 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw59s\" (UniqueName: \"kubernetes.io/projected/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-kube-api-access-lw59s\") pod \"maas-default-gateway-openshift-default-77f556d647-v8xs6\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.503949 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.503865 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:35.813693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.813617 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6"] Oct 30 09:12:35.817753 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:12:35.817719 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d8a515_9a87_4cf5_9ec3_08fda2172bcc.slice/crio-66a8bdbb4fc749459558e97e3d820ccc122638741dd0239904beb21660a3d012 WatchSource:0}: Error finding container 66a8bdbb4fc749459558e97e3d820ccc122638741dd0239904beb21660a3d012: Status 404 returned error can't find the container with id 66a8bdbb4fc749459558e97e3d820ccc122638741dd0239904beb21660a3d012 Oct 30 09:12:35.976284 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:35.976244 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" event={"ID":"26d8a515-9a87-4cf5-9ec3-08fda2172bcc","Type":"ContainerStarted","Data":"66a8bdbb4fc749459558e97e3d820ccc122638741dd0239904beb21660a3d012"} Oct 30 09:12:38.473980 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:38.473952 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:12:38.474248 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:38.474015 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:12:38.474248 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:38.474041 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:12:38.996280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:38.996231 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" event={"ID":"26d8a515-9a87-4cf5-9ec3-08fda2172bcc","Type":"ContainerStarted","Data":"8f1efd018c309b64127d8d4489a42098fb79ccf858b5f56aecdd6b4265d9c70a"} Oct 30 09:12:39.067495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:39.067433 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" podStartSLOduration=2.413916619 podStartE2EDuration="5.067415019s" totalImagesPullingTime="2.6534984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:12:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:12:39.065879845 +0000 UTC m=+1076.928585689" watchObservedRunningTime="2026-10-30 09:12:39.067415019 +0000 UTC m=+1076.930120841" Oct 30 09:12:39.504943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:39.504906 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:39.506284 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:39.506262 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v8xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 09:12:39.506353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:39.506309 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" podUID="26d8a515-9a87-4cf5-9ec3-08fda2172bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 09:12:40.504714 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:40.504674 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v8xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 09:12:40.504915 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:40.504747 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" podUID="26d8a515-9a87-4cf5-9ec3-08fda2172bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 09:12:41.204740 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:41.204700 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-844c8bbcf4-46cp6"] Oct 30 09:12:41.205198 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:41.204921 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-844c8bbcf4-46cp6" podUID="20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e" containerName="authorino" containerID="cri-o://54021f650cd7871512988d88245c2a2240abb6d758be895a0f235e363e0965dc" gracePeriod=30 Oct 30 09:12:41.503754 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:41.503718 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-844c8bbcf4-46cp6" Oct 30 09:12:41.504266 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:41.504237 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v8xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 09:12:41.504396 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:41.504286 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" podUID="26d8a515-9a87-4cf5-9ec3-08fda2172bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 09:12:41.563619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:41.563570 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e-kube-api-access-69nn7\" (UniqueName: \"kubernetes.io/projected/20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e-kube-api-access-69nn7\") pod \"20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e\" (UID: \"20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e\") " Oct 30 09:12:41.563831 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:41.563647 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e-tls-cert\" (UniqueName: \"kubernetes.io/secret/20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e-tls-cert\") pod \"20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e\" (UID: \"20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e\") " Oct 30 09:12:41.566132 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:41.566097 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e-kube-api-access-69nn7" pod "20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e" (UID: "20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e"). InnerVolumeSpecName "kube-api-access-69nn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:12:41.576450 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:41.576396 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e-tls-cert" pod "20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e" (UID: "20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:12:41.664670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:41.664622 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69nn7\" (UniqueName: \"kubernetes.io/projected/20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e-kube-api-access-69nn7\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:12:41.664670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:41.664665 5938 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e-tls-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:12:42.012793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:42.012758 5938 generic.go:356] "Generic (PLEG): container finished" podID="20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e" containerID="54021f650cd7871512988d88245c2a2240abb6d758be895a0f235e363e0965dc" exitCode=0 Oct 30 09:12:42.012997 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:42.012814 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-844c8bbcf4-46cp6" Oct 30 09:12:42.012997 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:42.012833 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-844c8bbcf4-46cp6" event={"ID":"20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e","Type":"ContainerDied","Data":"54021f650cd7871512988d88245c2a2240abb6d758be895a0f235e363e0965dc"} Oct 30 09:12:42.012997 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:42.012864 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-844c8bbcf4-46cp6" event={"ID":"20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e","Type":"ContainerDied","Data":"d4c54327b336f8186eaedc6fd53381e0e49e7d969eb933e932c387f9f887499a"} Oct 30 09:12:42.012997 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:42.012880 5938 scope.go:122] "RemoveContainer" containerID="54021f650cd7871512988d88245c2a2240abb6d758be895a0f235e363e0965dc" Oct 30 09:12:42.044813 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:42.044782 5938 scope.go:122] "RemoveContainer" containerID="54021f650cd7871512988d88245c2a2240abb6d758be895a0f235e363e0965dc" Oct 30 09:12:42.045490 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:12:42.045455 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54021f650cd7871512988d88245c2a2240abb6d758be895a0f235e363e0965dc\": container with ID starting with 54021f650cd7871512988d88245c2a2240abb6d758be895a0f235e363e0965dc not found: ID does not exist" containerID="54021f650cd7871512988d88245c2a2240abb6d758be895a0f235e363e0965dc" Oct 30 09:12:42.045652 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:42.045504 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54021f650cd7871512988d88245c2a2240abb6d758be895a0f235e363e0965dc"} err="failed to get container status \"54021f650cd7871512988d88245c2a2240abb6d758be895a0f235e363e0965dc\": rpc error: code = NotFound desc = could not find container \"54021f650cd7871512988d88245c2a2240abb6d758be895a0f235e363e0965dc\": container with ID starting with 54021f650cd7871512988d88245c2a2240abb6d758be895a0f235e363e0965dc not found: ID does not exist" Oct 30 09:12:42.088793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:42.088752 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-844c8bbcf4-46cp6"] Oct 30 09:12:42.092199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:42.092165 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-844c8bbcf4-46cp6"] Oct 30 09:12:42.254952 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:42.254909 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e" path="/var/lib/kubelet/pods/20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e/volumes" Oct 30 09:12:42.505271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:42.505232 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v8xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 09:12:42.505458 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:42.505290 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" podUID="26d8a515-9a87-4cf5-9ec3-08fda2172bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 09:12:43.504562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:43.504527 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v8xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 09:12:43.505027 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:43.504582 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" podUID="26d8a515-9a87-4cf5-9ec3-08fda2172bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 09:12:44.505182 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:44.505130 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v8xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 09:12:44.505759 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:44.505198 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" podUID="26d8a515-9a87-4cf5-9ec3-08fda2172bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 09:12:45.505258 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:45.505213 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v8xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 09:12:45.505703 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:45.505275 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" podUID="26d8a515-9a87-4cf5-9ec3-08fda2172bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 09:12:45.813869 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:12:45.813765 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:12:46.895252 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:46.895216 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:46.895799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:46.895702 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:46.896664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:46.896643 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:12:49.919107 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:49.919073 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46"] Oct 30 09:12:49.919641 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:49.919494 5938 cpu_manager.go:414] "containerMap: removing container" podUID="20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e" containerName="authorino" Oct 30 09:12:49.919641 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:49.919511 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e" containerName="authorino" Oct 30 09:12:49.919727 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:49.919656 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="20dbcd2f-a195-4b2b-bbd9-9dc79e3eeb1e" containerName="authorino" Oct 30 09:12:49.924753 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:49.924724 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:49.945515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:49.945484 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz"] Oct 30 09:12:49.949337 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:49.949309 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:49.961727 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:49.961681 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46"] Oct 30 09:12:49.977776 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:49.977745 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz"] Oct 30 09:12:50.040355 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.040319 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558ecb6f-0981-4211-8768-1c5de58945b8-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.040355 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.040361 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.040636 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.040383 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.040636 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.040465 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.040636 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.040546 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.040636 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.040623 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558ecb6f-0981-4211-8768-1c5de58945b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.040797 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.040655 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.040797 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.040695 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.040797 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.040737 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.040907 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.040807 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhkg8\" (UniqueName: \"kubernetes.io/projected/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-kube-api-access-vhkg8\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.040907 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.040842 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.040972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.040899 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.040972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.040930 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.041048 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.040975 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.041048 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.040993 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.041048 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.041017 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558ecb6f-0981-4211-8768-1c5de58945b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.041197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.041064 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jkkh\" (UniqueName: \"kubernetes.io/projected/558ecb6f-0981-4211-8768-1c5de58945b8-kube-api-access-5jkkh\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.041197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.041094 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.141911 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.141871 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.141911 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.141913 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.142186 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.141936 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.142186 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.141973 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558ecb6f-0981-4211-8768-1c5de58945b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.142186 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142149 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.142350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142204 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.142350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142238 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.142350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142267 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhkg8\" (UniqueName: \"kubernetes.io/projected/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-kube-api-access-vhkg8\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.142350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142307 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.142350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142334 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.142350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142346 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.142724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142365 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.142724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142392 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.142724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142406 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.142724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142453 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.142724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142499 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558ecb6f-0981-4211-8768-1c5de58945b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.142724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142524 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jkkh\" (UniqueName: \"kubernetes.io/projected/558ecb6f-0981-4211-8768-1c5de58945b8-kube-api-access-5jkkh\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.142724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142571 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.142724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142572 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.142724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142692 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558ecb6f-0981-4211-8768-1c5de58945b8-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.142724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142704 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.143210 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142731 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.143210 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142736 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.143210 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.142952 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.143210 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.143166 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.143395 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.143247 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558ecb6f-0981-4211-8768-1c5de58945b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.143395 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.143270 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.143479 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.143436 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.144945 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.144923 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558ecb6f-0981-4211-8768-1c5de58945b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.145454 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.145425 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.145559 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.145471 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.145754 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.145737 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.163569 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.163527 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558ecb6f-0981-4211-8768-1c5de58945b8-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.163804 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.163658 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.163804 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.163678 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhkg8\" (UniqueName: \"kubernetes.io/projected/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-kube-api-access-vhkg8\") pod \"maas-default-gateway-openshift-default-77f556d647-2hjgz\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.165387 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.165357 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jkkh\" (UniqueName: \"kubernetes.io/projected/558ecb6f-0981-4211-8768-1c5de58945b8-kube-api-access-5jkkh\") pod \"maas-default-gateway-openshift-default-77f556d647-4ln46\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.240123 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.240012 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:50.265877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.265833 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:50.565509 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.565386 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46"] Oct 30 09:12:50.568505 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:12:50.568475 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558ecb6f_0981_4211_8768_1c5de58945b8.slice/crio-9e5e20cb3139805c17101f685a4986e83f7661c345912f592cd86f0a04ada2ee WatchSource:0}: Error finding container 9e5e20cb3139805c17101f685a4986e83f7661c345912f592cd86f0a04ada2ee: Status 404 returned error can't find the container with id 9e5e20cb3139805c17101f685a4986e83f7661c345912f592cd86f0a04ada2ee Oct 30 09:12:50.570825 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.570792 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:12:50.570912 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.570880 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:12:50.570949 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.570911 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:12:50.609046 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.602083 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz"] Oct 30 09:12:50.609046 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.608974 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:12:50.609162 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.609109 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:12:50.609207 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:50.609186 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:12:51.084756 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:51.084721 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" event={"ID":"e1c99ffc-1e18-49c3-8327-dbcc3e939a95","Type":"ContainerStarted","Data":"108489d5d48e437df8af8ef7bb71e72c28368553390422c9f0846a9e7bae99c1"} Oct 30 09:12:51.085189 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:51.084816 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" event={"ID":"e1c99ffc-1e18-49c3-8327-dbcc3e939a95","Type":"ContainerStarted","Data":"71f5b9d726313ce0af28f9363a5285db0c93f93692e7c212efceb2a836ff8c81"} Oct 30 09:12:51.087271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:51.087235 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" event={"ID":"558ecb6f-0981-4211-8768-1c5de58945b8","Type":"ContainerStarted","Data":"82f80820a2c48ad871f7b239205c03b46d130d780b54cbfa5e0eefa9a839ecee"} Oct 30 09:12:51.087416 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:51.087298 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" event={"ID":"558ecb6f-0981-4211-8768-1c5de58945b8","Type":"ContainerStarted","Data":"9e5e20cb3139805c17101f685a4986e83f7661c345912f592cd86f0a04ada2ee"} Oct 30 09:12:51.184831 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:51.184770 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" podStartSLOduration=2.184750561 podStartE2EDuration="2.184750561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:12:51.184480104 +0000 UTC m=+1089.047185935" watchObservedRunningTime="2026-10-30 09:12:51.184750561 +0000 UTC m=+1089.047456383" Oct 30 09:12:51.185505 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:51.185460 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" podStartSLOduration=2.185448829 podStartE2EDuration="2.185448829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:12:51.136671075 +0000 UTC m=+1088.999376897" watchObservedRunningTime="2026-10-30 09:12:51.185448829 +0000 UTC m=+1089.048154650" Oct 30 09:12:51.242543 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:51.242505 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:51.242926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:51.242902 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4ln46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 09:12:51.243098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:51.243074 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" podUID="558ecb6f-0981-4211-8768-1c5de58945b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 09:12:51.269924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:51.269890 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:51.271348 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:51.271305 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2hjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 09:12:51.271503 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:51.271364 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" podUID="e1c99ffc-1e18-49c3-8327-dbcc3e939a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 09:12:52.244176 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:52.244071 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4ln46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 09:12:52.244893 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:52.244856 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" podUID="558ecb6f-0981-4211-8768-1c5de58945b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 09:12:52.268092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:52.267953 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2hjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 09:12:52.268092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:52.268011 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" podUID="e1c99ffc-1e18-49c3-8327-dbcc3e939a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 09:12:53.243104 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:53.243064 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4ln46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 09:12:53.243410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:53.243378 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" podUID="558ecb6f-0981-4211-8768-1c5de58945b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 09:12:53.269613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:53.268035 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2hjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 09:12:53.269613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:53.268094 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" podUID="e1c99ffc-1e18-49c3-8327-dbcc3e939a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 09:12:54.244531 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:54.244032 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4ln46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 09:12:54.244531 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:54.244097 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" podUID="558ecb6f-0981-4211-8768-1c5de58945b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 09:12:54.267851 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:54.267704 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2hjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 09:12:54.267851 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:54.267762 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" podUID="e1c99ffc-1e18-49c3-8327-dbcc3e939a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 09:12:55.243547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:55.243052 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4ln46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 09:12:55.243547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:55.243121 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" podUID="558ecb6f-0981-4211-8768-1c5de58945b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 09:12:55.268197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:55.268053 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2hjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 09:12:55.268197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:55.268116 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" podUID="e1c99ffc-1e18-49c3-8327-dbcc3e939a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 09:12:56.242616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:56.242066 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4ln46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 09:12:56.242616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:56.242133 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" podUID="558ecb6f-0981-4211-8768-1c5de58945b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 09:12:56.269237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:56.269007 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2hjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 09:12:56.269237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:56.269062 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" podUID="e1c99ffc-1e18-49c3-8327-dbcc3e939a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 09:12:57.241442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:57.241395 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4ln46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 09:12:57.241640 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:57.241469 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" podUID="558ecb6f-0981-4211-8768-1c5de58945b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 09:12:57.267162 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:57.267120 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2hjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 09:12:57.267349 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:57.267186 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" podUID="e1c99ffc-1e18-49c3-8327-dbcc3e939a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 09:12:58.241402 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:58.241353 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4ln46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 09:12:58.241933 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:58.241422 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" podUID="558ecb6f-0981-4211-8768-1c5de58945b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 09:12:58.266431 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:58.266389 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2hjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 09:12:58.266627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:58.266454 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" podUID="e1c99ffc-1e18-49c3-8327-dbcc3e939a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 09:12:59.245704 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:59.245667 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:59.246114 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:59.245968 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:59.246901 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:59.246882 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:12:59.270895 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:59.270863 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:59.271288 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:59.271267 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:12:59.272089 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:12:59.272068 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:13:15.813031 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:13:15.812990 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:13:17.674445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:17.674401 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt"] Oct 30 09:13:17.685826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:17.685790 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:17.698328 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:17.698278 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt"] Oct 30 09:13:17.710337 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:17.710302 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-70374d4e-serving-cert\"" Oct 30 09:13:17.825519 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:17.825475 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de129ce9-8f24-433b-ade5-8a159c1b96e1-openshift-service-ca\") pod \"maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt\" (UID: \"de129ce9-8f24-433b-ade5-8a159c1b96e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:17.825519 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:17.825524 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de129ce9-8f24-433b-ade5-8a159c1b96e1-metrics-certs\") pod \"maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt\" (UID: \"de129ce9-8f24-433b-ade5-8a159c1b96e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:17.825820 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:17.825639 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de129ce9-8f24-433b-ade5-8a159c1b96e1-maas-api-tls\") pod \"maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt\" (UID: \"de129ce9-8f24-433b-ade5-8a159c1b96e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:17.825820 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:17.825719 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqrhf\" (UniqueName: \"kubernetes.io/projected/de129ce9-8f24-433b-ade5-8a159c1b96e1-kube-api-access-zqrhf\") pod \"maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt\" (UID: \"de129ce9-8f24-433b-ade5-8a159c1b96e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:17.926535 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:17.926425 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de129ce9-8f24-433b-ade5-8a159c1b96e1-openshift-service-ca\") pod \"maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt\" (UID: \"de129ce9-8f24-433b-ade5-8a159c1b96e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:17.926535 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:17.926476 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de129ce9-8f24-433b-ade5-8a159c1b96e1-metrics-certs\") pod \"maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt\" (UID: \"de129ce9-8f24-433b-ade5-8a159c1b96e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:17.926535 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:17.926517 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de129ce9-8f24-433b-ade5-8a159c1b96e1-maas-api-tls\") pod \"maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt\" (UID: \"de129ce9-8f24-433b-ade5-8a159c1b96e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:17.926954 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:17.926544 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqrhf\" (UniqueName: \"kubernetes.io/projected/de129ce9-8f24-433b-ade5-8a159c1b96e1-kube-api-access-zqrhf\") pod \"maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt\" (UID: \"de129ce9-8f24-433b-ade5-8a159c1b96e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:17.927274 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:17.927246 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de129ce9-8f24-433b-ade5-8a159c1b96e1-openshift-service-ca\") pod \"maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt\" (UID: \"de129ce9-8f24-433b-ade5-8a159c1b96e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:17.929460 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:17.929425 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de129ce9-8f24-433b-ade5-8a159c1b96e1-maas-api-tls\") pod \"maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt\" (UID: \"de129ce9-8f24-433b-ade5-8a159c1b96e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:17.929460 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:17.929447 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de129ce9-8f24-433b-ade5-8a159c1b96e1-metrics-certs\") pod \"maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt\" (UID: \"de129ce9-8f24-433b-ade5-8a159c1b96e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:17.947196 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:17.947162 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqrhf\" (UniqueName: \"kubernetes.io/projected/de129ce9-8f24-433b-ade5-8a159c1b96e1-kube-api-access-zqrhf\") pod \"maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt\" (UID: \"de129ce9-8f24-433b-ade5-8a159c1b96e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:18.003275 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:18.003230 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:18.302614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:18.302531 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt"] Oct 30 09:13:18.305539 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:13:18.305511 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde129ce9_8f24_433b_ade5_8a159c1b96e1.slice/crio-9ecf8bb1ccf3a8bdb68d2d62de2762bf301dfc7a944bd1fcd25cdad501c33cf9 WatchSource:0}: Error finding container 9ecf8bb1ccf3a8bdb68d2d62de2762bf301dfc7a944bd1fcd25cdad501c33cf9: Status 404 returned error can't find the container with id 9ecf8bb1ccf3a8bdb68d2d62de2762bf301dfc7a944bd1fcd25cdad501c33cf9 Oct 30 09:13:19.229402 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:19.229346 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" event={"ID":"de129ce9-8f24-433b-ade5-8a159c1b96e1","Type":"ContainerStarted","Data":"b9b25ac34edf8251aebbea6551fc8c0717adce260957d4cbcad6177058f4f93c"} Oct 30 09:13:19.229866 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:19.229419 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" event={"ID":"de129ce9-8f24-433b-ade5-8a159c1b96e1","Type":"ContainerStarted","Data":"9ecf8bb1ccf3a8bdb68d2d62de2762bf301dfc7a944bd1fcd25cdad501c33cf9"} Oct 30 09:13:19.229866 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:19.229451 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:19.275343 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:19.275288 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" podStartSLOduration=1.8029715 podStartE2EDuration="2.275270139s" totalImagesPullingTime="472.298639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:19.274179182 +0000 UTC m=+1117.136885003" watchObservedRunningTime="2026-10-30 09:13:19.275270139 +0000 UTC m=+1117.137975961" Oct 30 09:13:25.239987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:25.239957 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:26.566550 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.566516 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn"] Oct 30 09:13:26.570504 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.570481 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:26.587257 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.587220 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-6cf9ebb1-serving-cert\"" Oct 30 09:13:26.608554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.603616 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn"] Oct 30 09:13:26.714049 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.714000 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8ktf\" (UniqueName: \"kubernetes.io/projected/dad5744c-1cad-451a-931e-c00f7482f73a-kube-api-access-f8ktf\") pod \"maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn\" (UID: \"dad5744c-1cad-451a-931e-c00f7482f73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:26.714249 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.714080 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad5744c-1cad-451a-931e-c00f7482f73a-openshift-service-ca\") pod \"maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn\" (UID: \"dad5744c-1cad-451a-931e-c00f7482f73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:26.714249 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.714168 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad5744c-1cad-451a-931e-c00f7482f73a-metrics-certs\") pod \"maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn\" (UID: \"dad5744c-1cad-451a-931e-c00f7482f73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:26.714249 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.714216 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dad5744c-1cad-451a-931e-c00f7482f73a-maas-api-tls\") pod \"maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn\" (UID: \"dad5744c-1cad-451a-931e-c00f7482f73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:26.815526 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.815482 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8ktf\" (UniqueName: \"kubernetes.io/projected/dad5744c-1cad-451a-931e-c00f7482f73a-kube-api-access-f8ktf\") pod \"maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn\" (UID: \"dad5744c-1cad-451a-931e-c00f7482f73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:26.815749 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.815542 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad5744c-1cad-451a-931e-c00f7482f73a-openshift-service-ca\") pod \"maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn\" (UID: \"dad5744c-1cad-451a-931e-c00f7482f73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:26.815749 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.815627 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad5744c-1cad-451a-931e-c00f7482f73a-metrics-certs\") pod \"maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn\" (UID: \"dad5744c-1cad-451a-931e-c00f7482f73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:26.815749 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.815689 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dad5744c-1cad-451a-931e-c00f7482f73a-maas-api-tls\") pod \"maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn\" (UID: \"dad5744c-1cad-451a-931e-c00f7482f73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:26.816426 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.816358 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad5744c-1cad-451a-931e-c00f7482f73a-openshift-service-ca\") pod \"maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn\" (UID: \"dad5744c-1cad-451a-931e-c00f7482f73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:26.818437 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.818382 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad5744c-1cad-451a-931e-c00f7482f73a-metrics-certs\") pod \"maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn\" (UID: \"dad5744c-1cad-451a-931e-c00f7482f73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:26.818437 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.818425 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dad5744c-1cad-451a-931e-c00f7482f73a-maas-api-tls\") pod \"maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn\" (UID: \"dad5744c-1cad-451a-931e-c00f7482f73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:26.837267 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.837225 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8ktf\" (UniqueName: \"kubernetes.io/projected/dad5744c-1cad-451a-931e-c00f7482f73a-kube-api-access-f8ktf\") pod \"maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn\" (UID: \"dad5744c-1cad-451a-931e-c00f7482f73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:26.884155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:26.884111 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:27.183452 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:27.183426 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn"] Oct 30 09:13:27.186503 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:13:27.186472 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad5744c_1cad_451a_931e_c00f7482f73a.slice/crio-a495170e00eb840b96541ab5d90d1b3a80a9487c31f77a613a8e7e27fbf5cf6f WatchSource:0}: Error finding container a495170e00eb840b96541ab5d90d1b3a80a9487c31f77a613a8e7e27fbf5cf6f: Status 404 returned error can't find the container with id a495170e00eb840b96541ab5d90d1b3a80a9487c31f77a613a8e7e27fbf5cf6f Oct 30 09:13:27.271696 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:27.271654 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" event={"ID":"dad5744c-1cad-451a-931e-c00f7482f73a","Type":"ContainerStarted","Data":"a495170e00eb840b96541ab5d90d1b3a80a9487c31f77a613a8e7e27fbf5cf6f"} Oct 30 09:13:28.277845 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:28.277809 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" event={"ID":"dad5744c-1cad-451a-931e-c00f7482f73a","Type":"ContainerStarted","Data":"239eebee0f703bdce29c83bd761303c2e9ad001d6b3f9eb8b1863058fbeeb363"} Oct 30 09:13:28.278284 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:28.277866 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:28.319663 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:28.319567 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" podStartSLOduration=2.026240838 podStartE2EDuration="2.319550595s" totalImagesPullingTime="293.309757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:28.318295114 +0000 UTC m=+1126.181000936" watchObservedRunningTime="2026-10-30 09:13:28.319550595 +0000 UTC m=+1126.182256415" Oct 30 09:13:34.288503 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:34.288476 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:45.809630 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:13:45.809503 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:13:49.466549 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:49.466508 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt"] Oct 30 09:13:49.467048 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:49.466851 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" podUID="de129ce9-8f24-433b-ade5-8a159c1b96e1" containerName="maas-api" containerID="cri-o://b9b25ac34edf8251aebbea6551fc8c0717adce260957d4cbcad6177058f4f93c" gracePeriod=30 Oct 30 09:13:49.731890 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:49.731809 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:49.858953 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:49.858920 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de129ce9-8f24-433b-ade5-8a159c1b96e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de129ce9-8f24-433b-ade5-8a159c1b96e1-openshift-service-ca\") pod \"de129ce9-8f24-433b-ade5-8a159c1b96e1\" (UID: \"de129ce9-8f24-433b-ade5-8a159c1b96e1\") " Oct 30 09:13:49.859139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:49.859060 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de129ce9-8f24-433b-ade5-8a159c1b96e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de129ce9-8f24-433b-ade5-8a159c1b96e1-maas-api-tls\") pod \"de129ce9-8f24-433b-ade5-8a159c1b96e1\" (UID: \"de129ce9-8f24-433b-ade5-8a159c1b96e1\") " Oct 30 09:13:49.859139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:49.859110 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de129ce9-8f24-433b-ade5-8a159c1b96e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de129ce9-8f24-433b-ade5-8a159c1b96e1-metrics-certs\") pod \"de129ce9-8f24-433b-ade5-8a159c1b96e1\" (UID: \"de129ce9-8f24-433b-ade5-8a159c1b96e1\") " Oct 30 09:13:49.859230 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:49.859146 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de129ce9-8f24-433b-ade5-8a159c1b96e1-kube-api-access-zqrhf\" (UniqueName: \"kubernetes.io/projected/de129ce9-8f24-433b-ade5-8a159c1b96e1-kube-api-access-zqrhf\") pod \"de129ce9-8f24-433b-ade5-8a159c1b96e1\" (UID: \"de129ce9-8f24-433b-ade5-8a159c1b96e1\") " Oct 30 09:13:49.859408 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:49.859378 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de129ce9-8f24-433b-ade5-8a159c1b96e1-openshift-service-ca" pod "de129ce9-8f24-433b-ade5-8a159c1b96e1" (UID: "de129ce9-8f24-433b-ade5-8a159c1b96e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:49.861837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:49.861799 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de129ce9-8f24-433b-ade5-8a159c1b96e1-metrics-certs" pod "de129ce9-8f24-433b-ade5-8a159c1b96e1" (UID: "de129ce9-8f24-433b-ade5-8a159c1b96e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:13:49.861968 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:49.861868 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de129ce9-8f24-433b-ade5-8a159c1b96e1-maas-api-tls" pod "de129ce9-8f24-433b-ade5-8a159c1b96e1" (UID: "de129ce9-8f24-433b-ade5-8a159c1b96e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:13:49.862096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:49.862073 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de129ce9-8f24-433b-ade5-8a159c1b96e1-kube-api-access-zqrhf" pod "de129ce9-8f24-433b-ade5-8a159c1b96e1" (UID: "de129ce9-8f24-433b-ade5-8a159c1b96e1"). InnerVolumeSpecName "kube-api-access-zqrhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:49.960994 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:49.960951 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de129ce9-8f24-433b-ade5-8a159c1b96e1-openshift-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:13:49.960994 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:49.960988 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de129ce9-8f24-433b-ade5-8a159c1b96e1-maas-api-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:13:49.960994 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:49.961001 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de129ce9-8f24-433b-ade5-8a159c1b96e1-metrics-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:13:49.961252 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:49.961012 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqrhf\" (UniqueName: \"kubernetes.io/projected/de129ce9-8f24-433b-ade5-8a159c1b96e1-kube-api-access-zqrhf\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:13:50.433654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:50.433609 5938 generic.go:356] "Generic (PLEG): container finished" podID="de129ce9-8f24-433b-ade5-8a159c1b96e1" containerID="b9b25ac34edf8251aebbea6551fc8c0717adce260957d4cbcad6177058f4f93c" exitCode=0 Oct 30 09:13:50.433654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:50.433650 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" event={"ID":"de129ce9-8f24-433b-ade5-8a159c1b96e1","Type":"ContainerDied","Data":"b9b25ac34edf8251aebbea6551fc8c0717adce260957d4cbcad6177058f4f93c"} Oct 30 09:13:50.433917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:50.433689 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" Oct 30 09:13:50.433917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:50.433710 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt" event={"ID":"de129ce9-8f24-433b-ade5-8a159c1b96e1","Type":"ContainerDied","Data":"9ecf8bb1ccf3a8bdb68d2d62de2762bf301dfc7a944bd1fcd25cdad501c33cf9"} Oct 30 09:13:50.433917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:50.433733 5938 scope.go:122] "RemoveContainer" containerID="b9b25ac34edf8251aebbea6551fc8c0717adce260957d4cbcad6177058f4f93c" Oct 30 09:13:50.443445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:50.443423 5938 scope.go:122] "RemoveContainer" containerID="b9b25ac34edf8251aebbea6551fc8c0717adce260957d4cbcad6177058f4f93c" Oct 30 09:13:50.443871 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:13:50.443844 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b25ac34edf8251aebbea6551fc8c0717adce260957d4cbcad6177058f4f93c\": container with ID starting with b9b25ac34edf8251aebbea6551fc8c0717adce260957d4cbcad6177058f4f93c not found: ID does not exist" containerID="b9b25ac34edf8251aebbea6551fc8c0717adce260957d4cbcad6177058f4f93c" Oct 30 09:13:50.443959 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:50.443886 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b25ac34edf8251aebbea6551fc8c0717adce260957d4cbcad6177058f4f93c"} err="failed to get container status \"b9b25ac34edf8251aebbea6551fc8c0717adce260957d4cbcad6177058f4f93c\": rpc error: code = NotFound desc = could not find container \"b9b25ac34edf8251aebbea6551fc8c0717adce260957d4cbcad6177058f4f93c\": container with ID starting with b9b25ac34edf8251aebbea6551fc8c0717adce260957d4cbcad6177058f4f93c not found: ID does not exist" Oct 30 09:13:50.489455 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:50.489414 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt"] Oct 30 09:13:50.503277 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:50.503240 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-70374d4e-7d845bdf58-r9hqt"] Oct 30 09:13:52.256429 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:52.256393 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de129ce9-8f24-433b-ade5-8a159c1b96e1" path="/var/lib/kubelet/pods/de129ce9-8f24-433b-ade5-8a159c1b96e1/volumes" Oct 30 09:13:59.514632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:59.514581 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn"] Oct 30 09:13:59.515141 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:59.514842 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" podUID="dad5744c-1cad-451a-931e-c00f7482f73a" containerName="maas-api" containerID="cri-o://239eebee0f703bdce29c83bd761303c2e9ad001d6b3f9eb8b1863058fbeeb363" gracePeriod=30 Oct 30 09:13:59.794965 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:59.794921 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:13:59.858272 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:59.858233 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad5744c-1cad-451a-931e-c00f7482f73a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad5744c-1cad-451a-931e-c00f7482f73a-metrics-certs\") pod \"dad5744c-1cad-451a-931e-c00f7482f73a\" (UID: \"dad5744c-1cad-451a-931e-c00f7482f73a\") " Oct 30 09:13:59.858487 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:59.858294 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad5744c-1cad-451a-931e-c00f7482f73a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad5744c-1cad-451a-931e-c00f7482f73a-openshift-service-ca\") pod \"dad5744c-1cad-451a-931e-c00f7482f73a\" (UID: \"dad5744c-1cad-451a-931e-c00f7482f73a\") " Oct 30 09:13:59.858487 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:59.858315 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad5744c-1cad-451a-931e-c00f7482f73a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dad5744c-1cad-451a-931e-c00f7482f73a-maas-api-tls\") pod \"dad5744c-1cad-451a-931e-c00f7482f73a\" (UID: \"dad5744c-1cad-451a-931e-c00f7482f73a\") " Oct 30 09:13:59.858487 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:59.858416 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad5744c-1cad-451a-931e-c00f7482f73a-kube-api-access-f8ktf\" (UniqueName: \"kubernetes.io/projected/dad5744c-1cad-451a-931e-c00f7482f73a-kube-api-access-f8ktf\") pod \"dad5744c-1cad-451a-931e-c00f7482f73a\" (UID: \"dad5744c-1cad-451a-931e-c00f7482f73a\") " Oct 30 09:13:59.858737 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:59.858710 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad5744c-1cad-451a-931e-c00f7482f73a-openshift-service-ca" pod "dad5744c-1cad-451a-931e-c00f7482f73a" (UID: "dad5744c-1cad-451a-931e-c00f7482f73a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:59.858899 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:59.858877 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad5744c-1cad-451a-931e-c00f7482f73a-openshift-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:13:59.861067 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:59.861033 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad5744c-1cad-451a-931e-c00f7482f73a-metrics-certs" pod "dad5744c-1cad-451a-931e-c00f7482f73a" (UID: "dad5744c-1cad-451a-931e-c00f7482f73a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:13:59.861183 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:59.861032 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad5744c-1cad-451a-931e-c00f7482f73a-kube-api-access-f8ktf" pod "dad5744c-1cad-451a-931e-c00f7482f73a" (UID: "dad5744c-1cad-451a-931e-c00f7482f73a"). InnerVolumeSpecName "kube-api-access-f8ktf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:59.861183 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:59.861093 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad5744c-1cad-451a-931e-c00f7482f73a-maas-api-tls" pod "dad5744c-1cad-451a-931e-c00f7482f73a" (UID: "dad5744c-1cad-451a-931e-c00f7482f73a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:13:59.960151 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:59.960116 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad5744c-1cad-451a-931e-c00f7482f73a-metrics-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:13:59.960151 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:59.960146 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dad5744c-1cad-451a-931e-c00f7482f73a-maas-api-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:13:59.960151 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:13:59.960155 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8ktf\" (UniqueName: \"kubernetes.io/projected/dad5744c-1cad-451a-931e-c00f7482f73a-kube-api-access-f8ktf\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:14:00.485446 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:00.485390 5938 generic.go:356] "Generic (PLEG): container finished" podID="dad5744c-1cad-451a-931e-c00f7482f73a" containerID="239eebee0f703bdce29c83bd761303c2e9ad001d6b3f9eb8b1863058fbeeb363" exitCode=0 Oct 30 09:14:00.485696 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:00.485457 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" event={"ID":"dad5744c-1cad-451a-931e-c00f7482f73a","Type":"ContainerDied","Data":"239eebee0f703bdce29c83bd761303c2e9ad001d6b3f9eb8b1863058fbeeb363"} Oct 30 09:14:00.485696 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:00.485561 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" event={"ID":"dad5744c-1cad-451a-931e-c00f7482f73a","Type":"ContainerDied","Data":"a495170e00eb840b96541ab5d90d1b3a80a9487c31f77a613a8e7e27fbf5cf6f"} Oct 30 09:14:00.485696 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:00.485626 5938 scope.go:122] "RemoveContainer" containerID="239eebee0f703bdce29c83bd761303c2e9ad001d6b3f9eb8b1863058fbeeb363" Oct 30 09:14:00.485876 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:00.485712 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn" Oct 30 09:14:00.497973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:00.497944 5938 scope.go:122] "RemoveContainer" containerID="239eebee0f703bdce29c83bd761303c2e9ad001d6b3f9eb8b1863058fbeeb363" Oct 30 09:14:00.498475 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:14:00.498450 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239eebee0f703bdce29c83bd761303c2e9ad001d6b3f9eb8b1863058fbeeb363\": container with ID starting with 239eebee0f703bdce29c83bd761303c2e9ad001d6b3f9eb8b1863058fbeeb363 not found: ID does not exist" containerID="239eebee0f703bdce29c83bd761303c2e9ad001d6b3f9eb8b1863058fbeeb363" Oct 30 09:14:00.498683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:00.498487 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239eebee0f703bdce29c83bd761303c2e9ad001d6b3f9eb8b1863058fbeeb363"} err="failed to get container status \"239eebee0f703bdce29c83bd761303c2e9ad001d6b3f9eb8b1863058fbeeb363\": rpc error: code = NotFound desc = could not find container \"239eebee0f703bdce29c83bd761303c2e9ad001d6b3f9eb8b1863058fbeeb363\": container with ID starting with 239eebee0f703bdce29c83bd761303c2e9ad001d6b3f9eb8b1863058fbeeb363 not found: ID does not exist" Oct 30 09:14:00.541632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:00.541517 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn"] Oct 30 09:14:00.555050 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:00.555008 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6cf9ebb1-7c74bd8fff-94lvn"] Oct 30 09:14:02.259342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:02.259309 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad5744c-1cad-451a-931e-c00f7482f73a" path="/var/lib/kubelet/pods/dad5744c-1cad-451a-931e-c00f7482f73a/volumes" Oct 30 09:14:03.462873 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.462832 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp"] Oct 30 09:14:03.463318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.463253 5938 cpu_manager.go:414] "containerMap: removing container" podUID="dad5744c-1cad-451a-931e-c00f7482f73a" containerName="maas-api" Oct 30 09:14:03.463318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.463266 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad5744c-1cad-451a-931e-c00f7482f73a" containerName="maas-api" Oct 30 09:14:03.463318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.463284 5938 cpu_manager.go:414] "containerMap: removing container" podUID="de129ce9-8f24-433b-ade5-8a159c1b96e1" containerName="maas-api" Oct 30 09:14:03.463318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.463291 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de129ce9-8f24-433b-ade5-8a159c1b96e1" containerName="maas-api" Oct 30 09:14:03.463476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.463388 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad5744c-1cad-451a-931e-c00f7482f73a" containerName="maas-api" Oct 30 09:14:03.463476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.463396 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="de129ce9-8f24-433b-ade5-8a159c1b96e1" containerName="maas-api" Oct 30 09:14:03.479879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.479846 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 09:14:03.488818 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.488786 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp"] Oct 30 09:14:03.502735 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.501393 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-3a61e1-serving-cert\"" Oct 30 09:14:03.604890 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.604843 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457441a6-59df-4b33-8675-55f9b5382fad-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp\" (UID: \"457441a6-59df-4b33-8675-55f9b5382fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 09:14:03.605104 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.604900 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd2mz\" (UniqueName: \"kubernetes.io/projected/457441a6-59df-4b33-8675-55f9b5382fad-kube-api-access-qd2mz\") pod \"maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp\" (UID: \"457441a6-59df-4b33-8675-55f9b5382fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 09:14:03.605183 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.605101 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457441a6-59df-4b33-8675-55f9b5382fad-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp\" (UID: \"457441a6-59df-4b33-8675-55f9b5382fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 09:14:03.605183 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.605150 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457441a6-59df-4b33-8675-55f9b5382fad-metrics-certs\") pod \"maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp\" (UID: \"457441a6-59df-4b33-8675-55f9b5382fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 09:14:03.706410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.706380 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457441a6-59df-4b33-8675-55f9b5382fad-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp\" (UID: \"457441a6-59df-4b33-8675-55f9b5382fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 09:14:03.706660 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.706418 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd2mz\" (UniqueName: \"kubernetes.io/projected/457441a6-59df-4b33-8675-55f9b5382fad-kube-api-access-qd2mz\") pod \"maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp\" (UID: \"457441a6-59df-4b33-8675-55f9b5382fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 09:14:03.706660 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.706475 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457441a6-59df-4b33-8675-55f9b5382fad-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp\" (UID: \"457441a6-59df-4b33-8675-55f9b5382fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 09:14:03.706660 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.706495 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457441a6-59df-4b33-8675-55f9b5382fad-metrics-certs\") pod \"maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp\" (UID: \"457441a6-59df-4b33-8675-55f9b5382fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 09:14:03.707214 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.707192 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457441a6-59df-4b33-8675-55f9b5382fad-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp\" (UID: \"457441a6-59df-4b33-8675-55f9b5382fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 09:14:03.709209 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.709186 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457441a6-59df-4b33-8675-55f9b5382fad-metrics-certs\") pod \"maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp\" (UID: \"457441a6-59df-4b33-8675-55f9b5382fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 09:14:03.709299 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.709255 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457441a6-59df-4b33-8675-55f9b5382fad-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp\" (UID: \"457441a6-59df-4b33-8675-55f9b5382fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 09:14:03.729702 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.729631 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd2mz\" (UniqueName: \"kubernetes.io/projected/457441a6-59df-4b33-8675-55f9b5382fad-kube-api-access-qd2mz\") pod \"maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp\" (UID: \"457441a6-59df-4b33-8675-55f9b5382fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 09:14:03.818781 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:03.818552 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 09:14:04.116497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:04.116402 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp"] Oct 30 09:14:04.120264 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:14:04.120234 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457441a6_59df_4b33_8675_55f9b5382fad.slice/crio-c7f7a463e96a55923594237494aae8cb6e20e1f98aa0dcda5026e69389c10638 WatchSource:0}: Error finding container c7f7a463e96a55923594237494aae8cb6e20e1f98aa0dcda5026e69389c10638: Status 404 returned error can't find the container with id c7f7a463e96a55923594237494aae8cb6e20e1f98aa0dcda5026e69389c10638 Oct 30 09:14:04.524278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:04.524240 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" event={"ID":"457441a6-59df-4b33-8675-55f9b5382fad","Type":"ContainerStarted","Data":"5bc7ca94fdbcefe4190bdd242b7f5294bc0c0d0484836c7c4b1e936283821adc"} Oct 30 09:14:04.524711 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:04.524301 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" event={"ID":"457441a6-59df-4b33-8675-55f9b5382fad","Type":"ContainerStarted","Data":"c7f7a463e96a55923594237494aae8cb6e20e1f98aa0dcda5026e69389c10638"} Oct 30 09:14:04.524711 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:04.524399 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 09:14:04.565780 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:04.565727 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" podStartSLOduration=1.249875755 podStartE2EDuration="1.565710773s" totalImagesPullingTime="315.835018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:14:04.564942953 +0000 UTC m=+1162.427648774" watchObservedRunningTime="2026-10-30 09:14:04.565710773 +0000 UTC m=+1162.428416588" Oct 30 09:14:10.535670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:10.535640 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 09:14:15.796995 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.796961 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl"] Oct 30 09:14:15.801065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.801036 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.812450 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:14:15.812417 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:14:15.832463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.832428 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-tmp-dir\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.832788 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.832746 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-home\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.832906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.832857 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7722f\" (UniqueName: \"kubernetes.io/projected/89552945-90ed-4c3c-b93a-a9e106aa4bf8-kube-api-access-7722f\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.832906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.832489 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2f69c8\"/\"openshift-service-ca.crt\"" Oct 30 09:14:15.832906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.832522 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2f69c8\"/\"simulator-w0-2f69c8-kserve-self-signed-certs\"" Oct 30 09:14:15.832906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.832644 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2f69c8\"/\"default-dockercfg-lh8pz\"" Oct 30 09:14:15.833086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.832489 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2f69c8\"/\"kube-root-ca.crt\"" Oct 30 09:14:15.833086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.833009 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-dshm\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.833086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.833064 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89552945-90ed-4c3c-b93a-a9e106aa4bf8-tls-certs\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.833210 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.833137 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-model-cache\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.834008 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.833987 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl"] Oct 30 09:14:15.934907 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.934865 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-dshm\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.935120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.934927 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89552945-90ed-4c3c-b93a-a9e106aa4bf8-tls-certs\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.935120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.934990 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-model-cache\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.935120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.935038 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-tmp-dir\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.935120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.935074 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-home\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.935349 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.935131 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7722f\" (UniqueName: \"kubernetes.io/projected/89552945-90ed-4c3c-b93a-a9e106aa4bf8-kube-api-access-7722f\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.935422 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.935397 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-model-cache\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.935495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.935435 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-tmp-dir\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.935766 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.935736 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-home\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.938118 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.938088 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-dshm\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.938458 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.938437 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89552945-90ed-4c3c-b93a-a9e106aa4bf8-tls-certs\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:15.955956 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:15.955918 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7722f\" (UniqueName: \"kubernetes.io/projected/89552945-90ed-4c3c-b93a-a9e106aa4bf8-kube-api-access-7722f\") pod \"simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:16.116314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:16.116285 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:16.416132 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:16.415988 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl"] Oct 30 09:14:16.418478 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:14:16.418438 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89552945_90ed_4c3c_b93a_a9e106aa4bf8.slice/crio-77589f64c9670e36df339afa410b92c9c4db2a7a8e47fbf683c8a6278edbf3bc WatchSource:0}: Error finding container 77589f64c9670e36df339afa410b92c9c4db2a7a8e47fbf683c8a6278edbf3bc: Status 404 returned error can't find the container with id 77589f64c9670e36df339afa410b92c9c4db2a7a8e47fbf683c8a6278edbf3bc Oct 30 09:14:16.592642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:16.592601 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" event={"ID":"89552945-90ed-4c3c-b93a-a9e106aa4bf8","Type":"ContainerStarted","Data":"77589f64c9670e36df339afa410b92c9c4db2a7a8e47fbf683c8a6278edbf3bc"} Oct 30 09:14:18.605825 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:18.605786 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" event={"ID":"89552945-90ed-4c3c-b93a-a9e106aa4bf8","Type":"ContainerStarted","Data":"dfbff12f5fe892fd3cf5a6e5e855ee130e7bab065cb0575801b4fd8ef1b2d99f"} Oct 30 09:14:18.649436 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:18.649380 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" podStartSLOduration=2.508529038 podStartE2EDuration="3.649366304s" totalImagesPullingTime="1.140837266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:14:18.649348955 +0000 UTC m=+1176.512054759" watchObservedRunningTime="2026-10-30 09:14:18.649366304 +0000 UTC m=+1176.512072125" Oct 30 09:14:19.116810 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:19.116765 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:25.970296 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:25.970209 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq"] Oct 30 09:14:25.974527 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:25.974501 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:25.992158 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:25.992126 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq"] Oct 30 09:14:26.004495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.004445 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3a61e1\"/\"openshift-service-ca.crt\"" Oct 30 09:14:26.004495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.004471 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3a61e1\"/\"kube-root-ca.crt\"" Oct 30 09:14:26.006893 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.006868 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3a61e1\"/\"simulator-w4-3a61e1-kserve-self-signed-certs\"" Oct 30 09:14:26.006960 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.006885 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3a61e1\"/\"default-dockercfg-tfdgb\"" Oct 30 09:14:26.038778 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.038741 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-home\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.038967 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.038802 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-tmp-dir\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.038967 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.038882 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-model-cache\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.038967 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.038937 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-dshm\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.039113 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.038983 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2371dfe4-c1c9-4773-b650-4d28e86a2f08-tls-certs\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.039184 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.039157 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldkxf\" (UniqueName: \"kubernetes.io/projected/2371dfe4-c1c9-4773-b650-4d28e86a2f08-kube-api-access-ldkxf\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.117451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.117409 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:26.136578 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.136547 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:26.140045 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.140016 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldkxf\" (UniqueName: \"kubernetes.io/projected/2371dfe4-c1c9-4773-b650-4d28e86a2f08-kube-api-access-ldkxf\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.140208 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.140094 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-home\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.140208 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.140164 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-tmp-dir\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.140410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.140218 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-model-cache\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.140410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.140247 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-dshm\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.140410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.140285 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2371dfe4-c1c9-4773-b650-4d28e86a2f08-tls-certs\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.140621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.140574 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-tmp-dir\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.140694 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.140620 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-home\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.140694 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.140645 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-model-cache\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.143001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.142968 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-dshm\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.143118 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.143105 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2371dfe4-c1c9-4773-b650-4d28e86a2f08-tls-certs\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.160716 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.160680 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldkxf\" (UniqueName: \"kubernetes.io/projected/2371dfe4-c1c9-4773-b650-4d28e86a2f08-kube-api-access-ldkxf\") pod \"simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.288262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.288167 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:26.587328 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.587276 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq"] Oct 30 09:14:26.590519 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:14:26.590489 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2371dfe4_c1c9_4773_b650_4d28e86a2f08.slice/crio-8b6b6e9fa64a79992932dd2d4e8fb12988ea773ea65177ccc5f4a4a5b570448a WatchSource:0}: Error finding container 8b6b6e9fa64a79992932dd2d4e8fb12988ea773ea65177ccc5f4a4a5b570448a: Status 404 returned error can't find the container with id 8b6b6e9fa64a79992932dd2d4e8fb12988ea773ea65177ccc5f4a4a5b570448a Oct 30 09:14:26.661319 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.661270 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" event={"ID":"2371dfe4-c1c9-4773-b650-4d28e86a2f08","Type":"ContainerStarted","Data":"223df8d825a01a7d078596148b24ddced81bdaad40c3f2cd9733a2179a3abe93"} Oct 30 09:14:26.661510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.661330 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" event={"ID":"2371dfe4-c1c9-4773-b650-4d28e86a2f08","Type":"ContainerStarted","Data":"8b6b6e9fa64a79992932dd2d4e8fb12988ea773ea65177ccc5f4a4a5b570448a"} Oct 30 09:14:26.673294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.673256 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:14:26.702274 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:26.702212 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podStartSLOduration=1.7021929249999999 podStartE2EDuration="1.702192925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:14:26.701232811 +0000 UTC m=+1184.563938635" watchObservedRunningTime="2026-10-30 09:14:26.702192925 +0000 UTC m=+1184.564898935" Oct 30 09:14:27.288778 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:27.288737 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:31.509710 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.509667 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz"] Oct 30 09:14:31.513502 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.513476 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.535096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.535062 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"openshift-service-ca.crt\"" Oct 30 09:14:31.549032 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.549002 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"kube-root-ca.crt\"" Oct 30 09:14:31.550319 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.550289 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz"] Oct 30 09:14:31.556502 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.556473 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"default-dockercfg-z2c5q\"" Oct 30 09:14:31.556697 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.556521 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"simulator-w3-176b6c-kserve-self-signed-certs\"" Oct 30 09:14:31.597009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.596964 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-dshm\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.597199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.597120 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k5z9\" (UniqueName: \"kubernetes.io/projected/f900f402-3458-4a8e-aa3b-a92b7ede222f-kube-api-access-8k5z9\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.597251 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.597192 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-model-cache\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.597251 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.597230 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-home\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.597325 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.597261 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f900f402-3458-4a8e-aa3b-a92b7ede222f-tls-certs\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.597325 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.597304 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-tmp-dir\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.699061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.699021 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-dshm\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.699277 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.699137 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k5z9\" (UniqueName: \"kubernetes.io/projected/f900f402-3458-4a8e-aa3b-a92b7ede222f-kube-api-access-8k5z9\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.699277 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.699176 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-model-cache\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.699277 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.699217 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-home\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.699277 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.699256 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f900f402-3458-4a8e-aa3b-a92b7ede222f-tls-certs\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.699502 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.699297 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-tmp-dir\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.699680 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.699645 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-model-cache\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.700166 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.700141 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-tmp-dir\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.700269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.700192 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-home\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.701775 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.701743 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-dshm\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.702167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.702141 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f900f402-3458-4a8e-aa3b-a92b7ede222f-tls-certs\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.721102 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.721061 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k5z9\" (UniqueName: \"kubernetes.io/projected/f900f402-3458-4a8e-aa3b-a92b7ede222f-kube-api-access-8k5z9\") pod \"simulator-w3-176b6c-kserve-86db7cfcf-xh6cz\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:31.831448 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:31.831351 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:32.136859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:32.136824 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz"] Oct 30 09:14:32.140342 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:14:32.140302 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf900f402_3458_4a8e_aa3b_a92b7ede222f.slice/crio-5ca18b98ef3cd46465d72c9808fd9cf065fb9c12cf5c8f86551803c83c9ad402 WatchSource:0}: Error finding container 5ca18b98ef3cd46465d72c9808fd9cf065fb9c12cf5c8f86551803c83c9ad402: Status 404 returned error can't find the container with id 5ca18b98ef3cd46465d72c9808fd9cf065fb9c12cf5c8f86551803c83c9ad402 Oct 30 09:14:32.695361 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:32.695312 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" event={"ID":"f900f402-3458-4a8e-aa3b-a92b7ede222f","Type":"ContainerStarted","Data":"37e85daa004f05bc9952aea7e81d120dcfcb632d1ce53bdae62cf3787b4186bb"} Oct 30 09:14:32.695361 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:32.695361 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" event={"ID":"f900f402-3458-4a8e-aa3b-a92b7ede222f","Type":"ContainerStarted","Data":"5ca18b98ef3cd46465d72c9808fd9cf065fb9c12cf5c8f86551803c83c9ad402"} Oct 30 09:14:32.735712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:32.735660 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" podStartSLOduration=1.7356456630000001 podStartE2EDuration="1.735645663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:14:32.734935262 +0000 UTC m=+1190.597641086" watchObservedRunningTime="2026-10-30 09:14:32.735645663 +0000 UTC m=+1190.598351483" Oct 30 09:14:32.832040 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:32.832005 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:36.289085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:36.289050 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:36.301633 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:36.301606 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:36.731326 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:36.731298 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:14:41.831730 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:41.831693 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:41.845029 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:41.844999 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:42.767423 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:42.767331 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:14:45.810045 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:14:45.810008 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:14:50.541860 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.541825 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh"] Oct 30 09:14:50.547755 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.547726 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.564491 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.564460 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh"] Oct 30 09:14:50.576666 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.576633 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2f69c8\"/\"premium-w0-2f69c8-kserve-self-signed-certs\"" Oct 30 09:14:50.604208 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.604174 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-home\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.604208 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.604213 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5nwm\" (UniqueName: \"kubernetes.io/projected/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-kube-api-access-q5nwm\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.604479 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.604335 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-model-cache\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.604479 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.604383 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-dshm\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.604479 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.604460 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-tmp-dir\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.604651 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.604510 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-tls-certs\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.705470 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.705434 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-home\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.705470 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.705479 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nwm\" (UniqueName: \"kubernetes.io/projected/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-kube-api-access-q5nwm\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.705852 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.705542 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-model-cache\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.705852 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.705620 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-dshm\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.705852 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.705650 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-tmp-dir\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.705852 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.705674 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-tls-certs\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.706070 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.705974 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-home\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.706127 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.706099 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-tmp-dir\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.706342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.706320 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-model-cache\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.708894 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.708860 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-dshm\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.709099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.709083 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-tls-certs\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.726494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.726456 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5nwm\" (UniqueName: \"kubernetes.io/projected/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-kube-api-access-q5nwm\") pod \"premium-w0-2f69c8-kserve-b987d7c4d-jc9xh\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:50.863186 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:50.863093 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:51.162007 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:51.161969 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh"] Oct 30 09:14:51.165348 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:14:51.165315 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15fcfc84_b3ad_4d47_93a9_1ab16e9f79fb.slice/crio-fb0099b738223af1496e69c19a05a4116fd81ca5f8d803cadbc0cd990250052e WatchSource:0}: Error finding container fb0099b738223af1496e69c19a05a4116fd81ca5f8d803cadbc0cd990250052e: Status 404 returned error can't find the container with id fb0099b738223af1496e69c19a05a4116fd81ca5f8d803cadbc0cd990250052e Oct 30 09:14:51.796702 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:51.796660 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" event={"ID":"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb","Type":"ContainerStarted","Data":"33a8a76f360a750f0f09a68b5271a41dd9f04914f405c274eab81e313410a901"} Oct 30 09:14:51.797108 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:51.796713 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" event={"ID":"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb","Type":"ContainerStarted","Data":"fb0099b738223af1496e69c19a05a4116fd81ca5f8d803cadbc0cd990250052e"} Oct 30 09:14:51.837088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:51.837028 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podStartSLOduration=1.837010458 podStartE2EDuration="1.837010458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:14:51.836471291 +0000 UTC m=+1209.699177113" watchObservedRunningTime="2026-10-30 09:14:51.837010458 +0000 UTC m=+1209.699716280" Oct 30 09:14:51.863894 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:51.863858 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:14:59.610499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.610463 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv"] Oct 30 09:14:59.614330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.614307 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.633723 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.633694 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv"] Oct 30 09:14:59.645627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.645560 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3a61e1\"/\"premium-w4-3a61e1-kserve-self-signed-certs\"" Oct 30 09:14:59.704464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.704425 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-dshm\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.704693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.704541 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-model-cache\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.704693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.704638 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-home\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.704693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.704671 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-tmp-dir\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.704871 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.704804 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372a7ce3-4393-48e0-a192-10e0530e933d-tls-certs\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.704871 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.704849 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvh9x\" (UniqueName: \"kubernetes.io/projected/372a7ce3-4393-48e0-a192-10e0530e933d-kube-api-access-vvh9x\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.806130 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.806094 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372a7ce3-4393-48e0-a192-10e0530e933d-tls-certs\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.806357 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.806140 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvh9x\" (UniqueName: \"kubernetes.io/projected/372a7ce3-4393-48e0-a192-10e0530e933d-kube-api-access-vvh9x\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.806357 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.806193 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-dshm\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.806357 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.806241 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-model-cache\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.806357 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.806275 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-home\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.806357 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.806312 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-tmp-dir\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.806784 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.806765 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-tmp-dir\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.807047 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.807027 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-model-cache\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.807686 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.807628 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-home\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.809841 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.809804 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-dshm\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.811169 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.811142 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372a7ce3-4393-48e0-a192-10e0530e933d-tls-certs\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.827602 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.827545 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvh9x\" (UniqueName: \"kubernetes.io/projected/372a7ce3-4393-48e0-a192-10e0530e933d-kube-api-access-vvh9x\") pod \"premium-w4-3a61e1-kserve-7b667c468c-6gdtv\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:14:59.931613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:14:59.931561 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:15:00.149759 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.149724 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk"] Oct 30 09:15:00.154100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.154072 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" Oct 30 09:15:00.166497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.166468 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb"] Oct 30 09:15:00.170206 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.170182 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" Oct 30 09:15:00.172523 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.172497 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:15:00.181918 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.181042 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk"] Oct 30 09:15:00.196622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.194832 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb"] Oct 30 09:15:00.214870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.214836 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc698a9a-0198-40f4-aca1-81a195c40b87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk\" (UID: \"dc698a9a-0198-40f4-aca1-81a195c40b87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" Oct 30 09:15:00.215083 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.214896 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlgbs\" (UniqueName: \"kubernetes.io/projected/3bc69d13-9b2a-4544-843b-fc575896a4b0-kube-api-access-hlgbs\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb\" (UID: \"3bc69d13-9b2a-4544-843b-fc575896a4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" Oct 30 09:15:00.215083 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.215016 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc69d13-9b2a-4544-843b-fc575896a4b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb\" (UID: \"3bc69d13-9b2a-4544-843b-fc575896a4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" Oct 30 09:15:00.215083 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.215053 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jg98\" (UniqueName: \"kubernetes.io/projected/dc698a9a-0198-40f4-aca1-81a195c40b87-kube-api-access-2jg98\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk\" (UID: \"dc698a9a-0198-40f4-aca1-81a195c40b87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" Oct 30 09:15:00.244507 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.244474 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv"] Oct 30 09:15:00.246954 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:15:00.246926 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod372a7ce3_4393_48e0_a192_10e0530e933d.slice/crio-270c69443b10343c19aac0beb25cc5cc557448c295b6c9514f45d100903f1411 WatchSource:0}: Error finding container 270c69443b10343c19aac0beb25cc5cc557448c295b6c9514f45d100903f1411: Status 404 returned error can't find the container with id 270c69443b10343c19aac0beb25cc5cc557448c295b6c9514f45d100903f1411 Oct 30 09:15:00.316706 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.316673 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlgbs\" (UniqueName: \"kubernetes.io/projected/3bc69d13-9b2a-4544-843b-fc575896a4b0-kube-api-access-hlgbs\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb\" (UID: \"3bc69d13-9b2a-4544-843b-fc575896a4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" Oct 30 09:15:00.316860 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.316730 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc69d13-9b2a-4544-843b-fc575896a4b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb\" (UID: \"3bc69d13-9b2a-4544-843b-fc575896a4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" Oct 30 09:15:00.316860 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.316764 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jg98\" (UniqueName: \"kubernetes.io/projected/dc698a9a-0198-40f4-aca1-81a195c40b87-kube-api-access-2jg98\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk\" (UID: \"dc698a9a-0198-40f4-aca1-81a195c40b87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" Oct 30 09:15:00.316984 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.316889 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc698a9a-0198-40f4-aca1-81a195c40b87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk\" (UID: \"dc698a9a-0198-40f4-aca1-81a195c40b87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" Oct 30 09:15:00.317629 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.317605 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc69d13-9b2a-4544-843b-fc575896a4b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb\" (UID: \"3bc69d13-9b2a-4544-843b-fc575896a4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" Oct 30 09:15:00.317629 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.317622 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc698a9a-0198-40f4-aca1-81a195c40b87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk\" (UID: \"dc698a9a-0198-40f4-aca1-81a195c40b87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" Oct 30 09:15:00.337397 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.337359 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jg98\" (UniqueName: \"kubernetes.io/projected/dc698a9a-0198-40f4-aca1-81a195c40b87-kube-api-access-2jg98\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk\" (UID: \"dc698a9a-0198-40f4-aca1-81a195c40b87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" Oct 30 09:15:00.337610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.337372 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlgbs\" (UniqueName: \"kubernetes.io/projected/3bc69d13-9b2a-4544-843b-fc575896a4b0-kube-api-access-hlgbs\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb\" (UID: \"3bc69d13-9b2a-4544-843b-fc575896a4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" Oct 30 09:15:00.469285 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.469198 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" Oct 30 09:15:00.489777 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.489743 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" Oct 30 09:15:00.767273 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.767251 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk"] Oct 30 09:15:00.769498 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:15:00.769464 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc698a9a_0198_40f4_aca1_81a195c40b87.slice/crio-52ab14b392630c7b163329c518c5468dacee331d217f6684acfa4affbaa7a704 WatchSource:0}: Error finding container 52ab14b392630c7b163329c518c5468dacee331d217f6684acfa4affbaa7a704: Status 404 returned error can't find the container with id 52ab14b392630c7b163329c518c5468dacee331d217f6684acfa4affbaa7a704 Oct 30 09:15:00.771450 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.771434 5938 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:15:00.799661 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.799547 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb"] Oct 30 09:15:00.802047 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:15:00.802014 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc69d13_9b2a_4544_843b_fc575896a4b0.slice/crio-7fec08fbb8114b0b786b33ff8574ef43fec7752090c3f296ebd55e83b2d1c6e4 WatchSource:0}: Error finding container 7fec08fbb8114b0b786b33ff8574ef43fec7752090c3f296ebd55e83b2d1c6e4: Status 404 returned error can't find the container with id 7fec08fbb8114b0b786b33ff8574ef43fec7752090c3f296ebd55e83b2d1c6e4 Oct 30 09:15:00.847674 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.847631 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" event={"ID":"dc698a9a-0198-40f4-aca1-81a195c40b87","Type":"ContainerStarted","Data":"52ab14b392630c7b163329c518c5468dacee331d217f6684acfa4affbaa7a704"} Oct 30 09:15:00.849395 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.849364 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" event={"ID":"372a7ce3-4393-48e0-a192-10e0530e933d","Type":"ContainerStarted","Data":"c5ed09cfe8fcd7902d2ebc569605df0885586deea90832a5b4584d3e55a607ac"} Oct 30 09:15:00.849535 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.849409 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" event={"ID":"372a7ce3-4393-48e0-a192-10e0530e933d","Type":"ContainerStarted","Data":"270c69443b10343c19aac0beb25cc5cc557448c295b6c9514f45d100903f1411"} Oct 30 09:15:00.850598 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.850562 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" event={"ID":"3bc69d13-9b2a-4544-843b-fc575896a4b0","Type":"ContainerStarted","Data":"7fec08fbb8114b0b786b33ff8574ef43fec7752090c3f296ebd55e83b2d1c6e4"} Oct 30 09:15:00.863924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.863846 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:15:00.877042 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.877011 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:15:00.890736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.890681 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podStartSLOduration=1.8906588439999998 podStartE2EDuration="1.890658844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:15:00.88890693 +0000 UTC m=+1218.751612748" watchObservedRunningTime="2026-10-30 09:15:00.890658844 +0000 UTC m=+1218.753364667" Oct 30 09:15:00.932445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:00.932412 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:15:01.879629 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:01.879571 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:15:03.880989 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:03.880948 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" event={"ID":"3bc69d13-9b2a-4544-843b-fc575896a4b0","Type":"ContainerStarted","Data":"b2098711d63b50829bc7b247e79edcbe50e48e678b4ab3b851b544fa057e0ce8"} Oct 30 09:15:03.882868 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:03.882840 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" event={"ID":"dc698a9a-0198-40f4-aca1-81a195c40b87","Type":"ContainerStarted","Data":"4842c432a5baea22ecfabf4be3f4052fbaee464db2b8eccc54d4a2c4baf3e596"} Oct 30 09:15:03.920051 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:03.920000 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" podStartSLOduration=1.234892766 podStartE2EDuration="3.919982868s" totalImagesPullingTime="2.685090102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:15:03.919339137 +0000 UTC m=+1221.782044960" watchObservedRunningTime="2026-10-30 09:15:03.919982868 +0000 UTC m=+1221.782688689" Oct 30 09:15:03.959163 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:03.959105 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" podStartSLOduration=1.63189907 podStartE2EDuration="3.959088853s" totalImagesPullingTime="2.327189783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:15:03.958131588 +0000 UTC m=+1221.820837408" watchObservedRunningTime="2026-10-30 09:15:03.959088853 +0000 UTC m=+1221.821794676" Oct 30 09:15:04.890121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:04.890083 5938 generic.go:356] "Generic (PLEG): container finished" podID="3bc69d13-9b2a-4544-843b-fc575896a4b0" containerID="b2098711d63b50829bc7b247e79edcbe50e48e678b4ab3b851b544fa057e0ce8" exitCode=60 Oct 30 09:15:04.890644 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:04.890174 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" event={"ID":"3bc69d13-9b2a-4544-843b-fc575896a4b0","Type":"ContainerDied","Data":"b2098711d63b50829bc7b247e79edcbe50e48e678b4ab3b851b544fa057e0ce8"} Oct 30 09:15:04.890644 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:04.890461 5938 scope.go:122] "RemoveContainer" containerID="b2098711d63b50829bc7b247e79edcbe50e48e678b4ab3b851b544fa057e0ce8" Oct 30 09:15:04.892040 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:04.892011 5938 generic.go:356] "Generic (PLEG): container finished" podID="dc698a9a-0198-40f4-aca1-81a195c40b87" containerID="4842c432a5baea22ecfabf4be3f4052fbaee464db2b8eccc54d4a2c4baf3e596" exitCode=60 Oct 30 09:15:04.892154 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:04.892039 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" event={"ID":"dc698a9a-0198-40f4-aca1-81a195c40b87","Type":"ContainerDied","Data":"4842c432a5baea22ecfabf4be3f4052fbaee464db2b8eccc54d4a2c4baf3e596"} Oct 30 09:15:04.892349 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:04.892330 5938 scope.go:122] "RemoveContainer" containerID="4842c432a5baea22ecfabf4be3f4052fbaee464db2b8eccc54d4a2c4baf3e596" Oct 30 09:15:05.899791 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:05.899759 5938 generic.go:356] "Generic (PLEG): container finished" podID="dc698a9a-0198-40f4-aca1-81a195c40b87" containerID="6f34b42503219cab047198ad94ff5b6d361abeba56136ca03a11abd8c7924195" exitCode=60 Oct 30 09:15:05.900230 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:05.899795 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" event={"ID":"dc698a9a-0198-40f4-aca1-81a195c40b87","Type":"ContainerDied","Data":"6f34b42503219cab047198ad94ff5b6d361abeba56136ca03a11abd8c7924195"} Oct 30 09:15:05.900230 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:05.899878 5938 scope.go:122] "RemoveContainer" containerID="4842c432a5baea22ecfabf4be3f4052fbaee464db2b8eccc54d4a2c4baf3e596" Oct 30 09:15:05.900230 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:05.900139 5938 scope.go:122] "RemoveContainer" containerID="6f34b42503219cab047198ad94ff5b6d361abeba56136ca03a11abd8c7924195" Oct 30 09:15:05.900461 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:15:05.900439 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk_odh-ai-gateway-infra(dc698a9a-0198-40f4-aca1-81a195c40b87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" podUID="dc698a9a-0198-40f4-aca1-81a195c40b87" Oct 30 09:15:05.902254 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:05.902233 5938 generic.go:356] "Generic (PLEG): container finished" podID="3bc69d13-9b2a-4544-843b-fc575896a4b0" containerID="99598ae319e2496efa8e7c98fbcb5eeab51a2db9054e841421004178ce49f9d3" exitCode=60 Oct 30 09:15:05.902352 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:05.902265 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" event={"ID":"3bc69d13-9b2a-4544-843b-fc575896a4b0","Type":"ContainerDied","Data":"99598ae319e2496efa8e7c98fbcb5eeab51a2db9054e841421004178ce49f9d3"} Oct 30 09:15:05.902536 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:05.902522 5938 scope.go:122] "RemoveContainer" containerID="99598ae319e2496efa8e7c98fbcb5eeab51a2db9054e841421004178ce49f9d3" Oct 30 09:15:05.902791 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:15:05.902774 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb_odh-ai-gateway-infra(3bc69d13-9b2a-4544-843b-fc575896a4b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" podUID="3bc69d13-9b2a-4544-843b-fc575896a4b0" Oct 30 09:15:05.948030 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:05.948010 5938 scope.go:122] "RemoveContainer" containerID="b2098711d63b50829bc7b247e79edcbe50e48e678b4ab3b851b544fa057e0ce8" Oct 30 09:15:06.910526 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:06.910408 5938 scope.go:122] "RemoveContainer" containerID="6f34b42503219cab047198ad94ff5b6d361abeba56136ca03a11abd8c7924195" Oct 30 09:15:06.911010 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:15:06.910766 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk_odh-ai-gateway-infra(dc698a9a-0198-40f4-aca1-81a195c40b87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" podUID="dc698a9a-0198-40f4-aca1-81a195c40b87" Oct 30 09:15:06.913729 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:06.913704 5938 scope.go:122] "RemoveContainer" containerID="99598ae319e2496efa8e7c98fbcb5eeab51a2db9054e841421004178ce49f9d3" Oct 30 09:15:06.913974 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:15:06.913954 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb_odh-ai-gateway-infra(3bc69d13-9b2a-4544-843b-fc575896a4b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" podUID="3bc69d13-9b2a-4544-843b-fc575896a4b0" Oct 30 09:15:07.293504 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.293402 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7"] Oct 30 09:15:07.299248 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.299217 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.314829 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.314795 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7"] Oct 30 09:15:07.318422 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.318389 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e3d2ff\"/\"default-dockercfg-jsvq8\"" Oct 30 09:15:07.318422 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.318393 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e3d2ff\"/\"openshift-service-ca.crt\"" Oct 30 09:15:07.318662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.318429 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e3d2ff\"/\"premium-w2-e3d2ff-kserve-self-signed-certs\"" Oct 30 09:15:07.318662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.318438 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e3d2ff\"/\"kube-root-ca.crt\"" Oct 30 09:15:07.406152 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.406101 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-dshm\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.406435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.406333 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-model-cache\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.406435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.406377 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-tmp-dir\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.406559 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.406528 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac81131-bcdc-452b-a551-dea9888c9155-tls-certs\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.406652 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.406564 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbx4j\" (UniqueName: \"kubernetes.io/projected/4ac81131-bcdc-452b-a551-dea9888c9155-kube-api-access-lbx4j\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.406652 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.406631 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-home\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.507481 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.507441 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-model-cache\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.507481 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.507478 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-tmp-dir\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.507779 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.507503 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac81131-bcdc-452b-a551-dea9888c9155-tls-certs\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.507779 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.507521 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbx4j\" (UniqueName: \"kubernetes.io/projected/4ac81131-bcdc-452b-a551-dea9888c9155-kube-api-access-lbx4j\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.507779 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.507564 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-home\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.507779 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.507671 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-dshm\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.508020 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.507950 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-tmp-dir\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.508074 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.508016 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-model-cache\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.508131 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.508121 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-home\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.510264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.510232 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-dshm\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.510418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.510392 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac81131-bcdc-452b-a551-dea9888c9155-tls-certs\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.528299 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.528253 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbx4j\" (UniqueName: \"kubernetes.io/projected/4ac81131-bcdc-452b-a551-dea9888c9155-kube-api-access-lbx4j\") pod \"premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.614369 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.614343 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:07.921721 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:07.921630 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7"] Oct 30 09:15:07.924454 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:15:07.924424 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ac81131_bcdc_452b_a551_dea9888c9155.slice/crio-29bfe7e0d3989cef3e79f0a459db473ec70a178ce071663831f70418449cfa52 WatchSource:0}: Error finding container 29bfe7e0d3989cef3e79f0a459db473ec70a178ce071663831f70418449cfa52: Status 404 returned error can't find the container with id 29bfe7e0d3989cef3e79f0a459db473ec70a178ce071663831f70418449cfa52 Oct 30 09:15:08.929167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:08.929121 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" event={"ID":"4ac81131-bcdc-452b-a551-dea9888c9155","Type":"ContainerStarted","Data":"10f6010d8d7837e4792ac4bcd500fbe263d6fc2c478cee785225a7ad5296278b"} Oct 30 09:15:08.929707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:08.929179 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" event={"ID":"4ac81131-bcdc-452b-a551-dea9888c9155","Type":"ContainerStarted","Data":"29bfe7e0d3989cef3e79f0a459db473ec70a178ce071663831f70418449cfa52"} Oct 30 09:15:08.969777 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:08.969702 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podStartSLOduration=1.969681695 podStartE2EDuration="1.969681695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:15:08.968275491 +0000 UTC m=+1226.830981315" watchObservedRunningTime="2026-10-30 09:15:08.969681695 +0000 UTC m=+1226.832387528" Oct 30 09:15:09.615237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:09.615202 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:09.931931 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:09.931837 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:15:09.945065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:09.945027 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:15:10.944418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:10.944383 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:15:11.282501 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.282419 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr"] Oct 30 09:15:11.286401 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.286375 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.305651 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.305613 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr"] Oct 30 09:15:11.316922 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.316883 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"premium-w3-176b6c-kserve-self-signed-certs\"" Oct 30 09:15:11.452539 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.452494 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-home\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.452745 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.452552 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-model-cache\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.452801 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.452748 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-tmp-dir\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.452842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.452799 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-dshm\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.452842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.452828 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ded0c49-3011-4730-af91-f1700a3034b3-tls-certs\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.452961 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.452935 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxn8p\" (UniqueName: \"kubernetes.io/projected/8ded0c49-3011-4730-af91-f1700a3034b3-kube-api-access-vxn8p\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.553760 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.553659 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxn8p\" (UniqueName: \"kubernetes.io/projected/8ded0c49-3011-4730-af91-f1700a3034b3-kube-api-access-vxn8p\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.553962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.553771 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-home\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.553962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.553798 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-model-cache\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.553962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.553888 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-tmp-dir\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.553962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.553938 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-dshm\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.554180 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.553967 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ded0c49-3011-4730-af91-f1700a3034b3-tls-certs\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.554238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.554205 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-home\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.554346 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.554322 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-model-cache\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.554408 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.554360 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-tmp-dir\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.556816 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.556784 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-dshm\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.556987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.556881 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ded0c49-3011-4730-af91-f1700a3034b3-tls-certs\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.574698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.574663 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxn8p\" (UniqueName: \"kubernetes.io/projected/8ded0c49-3011-4730-af91-f1700a3034b3-kube-api-access-vxn8p\") pod \"premium-w3-176b6c-kserve-c889c986-jw6fr\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.601999 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.601962 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:11.907445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.907409 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr"] Oct 30 09:15:11.910785 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:15:11.910751 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ded0c49_3011_4730_af91_f1700a3034b3.slice/crio-44acef725c84293227c0a78e3575ba2ec6b1c4a8a7fe1012a274bd5957f0da64 WatchSource:0}: Error finding container 44acef725c84293227c0a78e3575ba2ec6b1c4a8a7fe1012a274bd5957f0da64: Status 404 returned error can't find the container with id 44acef725c84293227c0a78e3575ba2ec6b1c4a8a7fe1012a274bd5957f0da64 Oct 30 09:15:11.946850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:11.946806 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" event={"ID":"8ded0c49-3011-4730-af91-f1700a3034b3","Type":"ContainerStarted","Data":"44acef725c84293227c0a78e3575ba2ec6b1c4a8a7fe1012a274bd5957f0da64"} Oct 30 09:15:12.955563 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:12.955529 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" event={"ID":"8ded0c49-3011-4730-af91-f1700a3034b3","Type":"ContainerStarted","Data":"c0c5cae79b865713f96c761b3eb1c9f428dadaec9cd095468cb1fe75fd9d9471"} Oct 30 09:15:12.996656 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:12.996565 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podStartSLOduration=1.996548839 podStartE2EDuration="1.996548839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:15:12.99451058 +0000 UTC m=+1230.857216402" watchObservedRunningTime="2026-10-30 09:15:12.996548839 +0000 UTC m=+1230.859254660" Oct 30 09:15:13.602777 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:13.602744 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:15.296640 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.296602 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp"] Oct 30 09:15:15.301106 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.301058 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.331310 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.331272 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c5447d\"/\"default-dockercfg-9qf68\"" Oct 30 09:15:15.331310 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.331305 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c5447d\"/\"kube-root-ca.crt\"" Oct 30 09:15:15.331570 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.331336 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c5447d\"/\"premium-w1-c5447d-kserve-self-signed-certs\"" Oct 30 09:15:15.331570 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.331430 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c5447d\"/\"openshift-service-ca.crt\"" Oct 30 09:15:15.333807 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.333766 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp"] Oct 30 09:15:15.396138 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.396094 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-model-cache\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.396336 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.396168 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-954gl\" (UniqueName: \"kubernetes.io/projected/0aa496ad-244c-437e-aeba-630c32464259-kube-api-access-954gl\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.396336 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.396304 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-dshm\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.396414 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.396346 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-tmp-dir\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.396573 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.396548 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aa496ad-244c-437e-aeba-630c32464259-tls-certs\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.396648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.396633 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-home\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.498187 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.498136 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aa496ad-244c-437e-aeba-630c32464259-tls-certs\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.498187 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.498182 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-home\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.498497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.498319 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-model-cache\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.498497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.498400 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-954gl\" (UniqueName: \"kubernetes.io/projected/0aa496ad-244c-437e-aeba-630c32464259-kube-api-access-954gl\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.498497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.498456 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-dshm\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.498497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.498484 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-tmp-dir\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.498740 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.498655 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-home\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.498740 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.498701 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-model-cache\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.498947 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.498924 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-tmp-dir\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.501172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.501138 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-dshm\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.501431 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.501409 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aa496ad-244c-437e-aeba-630c32464259-tls-certs\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.520055 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.520018 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-954gl\" (UniqueName: \"kubernetes.io/projected/0aa496ad-244c-437e-aeba-630c32464259-kube-api-access-954gl\") pod \"premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.616936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.616901 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:15.811412 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:15:15.811377 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:15:15.922146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.922127 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp"] Oct 30 09:15:15.924541 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:15:15.924516 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0aa496ad_244c_437e_aeba_630c32464259.slice/crio-34040528e097241820cccaf1dd92dbf370c1757a595257a046e2977d4932343c WatchSource:0}: Error finding container 34040528e097241820cccaf1dd92dbf370c1757a595257a046e2977d4932343c: Status 404 returned error can't find the container with id 34040528e097241820cccaf1dd92dbf370c1757a595257a046e2977d4932343c Oct 30 09:15:15.973185 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:15.973150 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" event={"ID":"0aa496ad-244c-437e-aeba-630c32464259","Type":"ContainerStarted","Data":"34040528e097241820cccaf1dd92dbf370c1757a595257a046e2977d4932343c"} Oct 30 09:15:16.979783 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:16.979746 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" event={"ID":"0aa496ad-244c-437e-aeba-630c32464259","Type":"ContainerStarted","Data":"d4ca7ff0509f3a5575c9dd9d60c5702ca14d8f2ee08c5ca6e52c9610db592b09"} Oct 30 09:15:17.020379 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:17.020330 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" podStartSLOduration=2.020316325 podStartE2EDuration="2.020316325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:15:17.020044882 +0000 UTC m=+1234.882750715" watchObservedRunningTime="2026-10-30 09:15:17.020316325 +0000 UTC m=+1234.883022145" Oct 30 09:15:17.614849 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:17.614811 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:17.617712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:17.617678 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:17.628578 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:17.628545 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:17.996217 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:17.996125 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:15:18.250707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:18.250623 5938 scope.go:122] "RemoveContainer" containerID="99598ae319e2496efa8e7c98fbcb5eeab51a2db9054e841421004178ce49f9d3" Oct 30 09:15:18.992153 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:18.992117 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" event={"ID":"3bc69d13-9b2a-4544-843b-fc575896a4b0","Type":"ContainerStarted","Data":"05ab82c09fc6e0e7465b89cc683de15a2cd2e7eef070bdedab855330c7b37709"} Oct 30 09:15:19.250436 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:19.250352 5938 scope.go:122] "RemoveContainer" containerID="6f34b42503219cab047198ad94ff5b6d361abeba56136ca03a11abd8c7924195" Oct 30 09:15:20.000422 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:20.000389 5938 generic.go:356] "Generic (PLEG): container finished" podID="3bc69d13-9b2a-4544-843b-fc575896a4b0" containerID="05ab82c09fc6e0e7465b89cc683de15a2cd2e7eef070bdedab855330c7b37709" exitCode=60 Oct 30 09:15:20.000653 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:20.000467 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" event={"ID":"3bc69d13-9b2a-4544-843b-fc575896a4b0","Type":"ContainerDied","Data":"05ab82c09fc6e0e7465b89cc683de15a2cd2e7eef070bdedab855330c7b37709"} Oct 30 09:15:20.000653 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:20.000529 5938 scope.go:122] "RemoveContainer" containerID="99598ae319e2496efa8e7c98fbcb5eeab51a2db9054e841421004178ce49f9d3" Oct 30 09:15:20.000933 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:20.000895 5938 scope.go:122] "RemoveContainer" containerID="05ab82c09fc6e0e7465b89cc683de15a2cd2e7eef070bdedab855330c7b37709" Oct 30 09:15:20.001216 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:15:20.001191 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb_odh-ai-gateway-infra(3bc69d13-9b2a-4544-843b-fc575896a4b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" podUID="3bc69d13-9b2a-4544-843b-fc575896a4b0" Oct 30 09:15:20.002635 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:20.002612 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" event={"ID":"dc698a9a-0198-40f4-aca1-81a195c40b87","Type":"ContainerStarted","Data":"b2828b11ef05fe6a50e1fcd8694bb3ac2f91ee18db88ded84d576bd4bd87a924"} Oct 30 09:15:20.041484 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:20.041455 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb"] Oct 30 09:15:21.010466 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:21.010434 5938 generic.go:356] "Generic (PLEG): container finished" podID="dc698a9a-0198-40f4-aca1-81a195c40b87" containerID="b2828b11ef05fe6a50e1fcd8694bb3ac2f91ee18db88ded84d576bd4bd87a924" exitCode=60 Oct 30 09:15:21.010939 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:21.010508 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" event={"ID":"dc698a9a-0198-40f4-aca1-81a195c40b87","Type":"ContainerDied","Data":"b2828b11ef05fe6a50e1fcd8694bb3ac2f91ee18db88ded84d576bd4bd87a924"} Oct 30 09:15:21.010939 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:21.010571 5938 scope.go:122] "RemoveContainer" containerID="6f34b42503219cab047198ad94ff5b6d361abeba56136ca03a11abd8c7924195" Oct 30 09:15:21.011018 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:21.011003 5938 scope.go:122] "RemoveContainer" containerID="b2828b11ef05fe6a50e1fcd8694bb3ac2f91ee18db88ded84d576bd4bd87a924" Oct 30 09:15:21.011343 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:15:21.011320 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk_odh-ai-gateway-infra(dc698a9a-0198-40f4-aca1-81a195c40b87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" podUID="dc698a9a-0198-40f4-aca1-81a195c40b87" Oct 30 09:15:21.108165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:21.108124 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk"] Oct 30 09:15:21.188569 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:21.188538 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" Oct 30 09:15:21.266407 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:21.266311 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc69d13-9b2a-4544-843b-fc575896a4b0-kube-api-access-hlgbs\" (UniqueName: \"kubernetes.io/projected/3bc69d13-9b2a-4544-843b-fc575896a4b0-kube-api-access-hlgbs\") pod \"3bc69d13-9b2a-4544-843b-fc575896a4b0\" (UID: \"3bc69d13-9b2a-4544-843b-fc575896a4b0\") " Oct 30 09:15:21.266407 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:21.266408 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc69d13-9b2a-4544-843b-fc575896a4b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc69d13-9b2a-4544-843b-fc575896a4b0-maas-api-service-ca\") pod \"3bc69d13-9b2a-4544-843b-fc575896a4b0\" (UID: \"3bc69d13-9b2a-4544-843b-fc575896a4b0\") " Oct 30 09:15:21.266861 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:21.266836 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc69d13-9b2a-4544-843b-fc575896a4b0-maas-api-service-ca" pod "3bc69d13-9b2a-4544-843b-fc575896a4b0" (UID: "3bc69d13-9b2a-4544-843b-fc575896a4b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:15:21.268935 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:21.268896 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc69d13-9b2a-4544-843b-fc575896a4b0-kube-api-access-hlgbs" pod "3bc69d13-9b2a-4544-843b-fc575896a4b0" (UID: "3bc69d13-9b2a-4544-843b-fc575896a4b0"). InnerVolumeSpecName "kube-api-access-hlgbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:15:21.368486 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:21.368444 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlgbs\" (UniqueName: \"kubernetes.io/projected/3bc69d13-9b2a-4544-843b-fc575896a4b0-kube-api-access-hlgbs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:15:21.368486 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:21.368482 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc69d13-9b2a-4544-843b-fc575896a4b0-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:15:21.602335 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:21.602243 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:21.615057 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:21.615027 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:22.021215 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:22.021175 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" event={"ID":"3bc69d13-9b2a-4544-843b-fc575896a4b0","Type":"ContainerDied","Data":"7fec08fbb8114b0b786b33ff8574ef43fec7752090c3f296ebd55e83b2d1c6e4"} Oct 30 09:15:22.021623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:22.021252 5938 scope.go:122] "RemoveContainer" containerID="05ab82c09fc6e0e7465b89cc683de15a2cd2e7eef070bdedab855330c7b37709" Oct 30 09:15:22.021623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:22.021194 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb" Oct 30 09:15:22.036796 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:22.036771 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:15:22.098845 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:22.098813 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb"] Oct 30 09:15:22.100445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:22.100417 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889195-8p4wb"] Oct 30 09:15:22.194720 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:22.194693 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" Oct 30 09:15:22.267830 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:22.267782 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc69d13-9b2a-4544-843b-fc575896a4b0" path="/var/lib/kubelet/pods/3bc69d13-9b2a-4544-843b-fc575896a4b0/volumes" Oct 30 09:15:22.278786 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:22.278686 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc698a9a-0198-40f4-aca1-81a195c40b87-kube-api-access-2jg98\" (UniqueName: \"kubernetes.io/projected/dc698a9a-0198-40f4-aca1-81a195c40b87-kube-api-access-2jg98\") pod \"dc698a9a-0198-40f4-aca1-81a195c40b87\" (UID: \"dc698a9a-0198-40f4-aca1-81a195c40b87\") " Oct 30 09:15:22.278952 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:22.278812 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc698a9a-0198-40f4-aca1-81a195c40b87-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc698a9a-0198-40f4-aca1-81a195c40b87-maas-api-service-ca\") pod \"dc698a9a-0198-40f4-aca1-81a195c40b87\" (UID: \"dc698a9a-0198-40f4-aca1-81a195c40b87\") " Oct 30 09:15:22.279270 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:22.279240 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc698a9a-0198-40f4-aca1-81a195c40b87-maas-api-service-ca" pod "dc698a9a-0198-40f4-aca1-81a195c40b87" (UID: "dc698a9a-0198-40f4-aca1-81a195c40b87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:15:22.281201 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:22.281166 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc698a9a-0198-40f4-aca1-81a195c40b87-kube-api-access-2jg98" pod "dc698a9a-0198-40f4-aca1-81a195c40b87" (UID: "dc698a9a-0198-40f4-aca1-81a195c40b87"). InnerVolumeSpecName "kube-api-access-2jg98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:15:22.380244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:22.380194 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jg98\" (UniqueName: \"kubernetes.io/projected/dc698a9a-0198-40f4-aca1-81a195c40b87-kube-api-access-2jg98\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:15:22.380244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:22.380241 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc698a9a-0198-40f4-aca1-81a195c40b87-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:15:23.029977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:23.029936 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" Oct 30 09:15:23.030487 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:23.029936 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk" event={"ID":"dc698a9a-0198-40f4-aca1-81a195c40b87","Type":"ContainerDied","Data":"52ab14b392630c7b163329c518c5468dacee331d217f6684acfa4affbaa7a704"} Oct 30 09:15:23.030487 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:23.030109 5938 scope.go:122] "RemoveContainer" containerID="b2828b11ef05fe6a50e1fcd8694bb3ac2f91ee18db88ded84d576bd4bd87a924" Oct 30 09:15:23.085162 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:23.085124 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk"] Oct 30 09:15:23.087846 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:23.087817 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889195-hs8fk"] Oct 30 09:15:24.257238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:24.257202 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc698a9a-0198-40f4-aca1-81a195c40b87" path="/var/lib/kubelet/pods/dc698a9a-0198-40f4-aca1-81a195c40b87/volumes" Oct 30 09:15:25.617167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:25.617128 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:25.630384 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:25.630357 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:26.059140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:15:26.059058 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:15:45.811083 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:15:45.811039 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:16:15.812290 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:16:15.812202 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:16:33.672832 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.672788 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf"] Oct 30 09:16:33.673523 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.673503 5938 cpu_manager.go:414] "containerMap: removing container" podUID="dc698a9a-0198-40f4-aca1-81a195c40b87" containerName="cleanup" Oct 30 09:16:33.673569 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.673526 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc698a9a-0198-40f4-aca1-81a195c40b87" containerName="cleanup" Oct 30 09:16:33.673569 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.673547 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3bc69d13-9b2a-4544-843b-fc575896a4b0" containerName="cleanup" Oct 30 09:16:33.673569 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.673555 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc69d13-9b2a-4544-843b-fc575896a4b0" containerName="cleanup" Oct 30 09:16:33.673824 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.673569 5938 cpu_manager.go:414] "containerMap: removing container" podUID="dc698a9a-0198-40f4-aca1-81a195c40b87" containerName="cleanup" Oct 30 09:16:33.673824 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.673578 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc698a9a-0198-40f4-aca1-81a195c40b87" containerName="cleanup" Oct 30 09:16:33.673824 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.673713 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3bc69d13-9b2a-4544-843b-fc575896a4b0" containerName="cleanup" Oct 30 09:16:33.673824 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.673724 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc69d13-9b2a-4544-843b-fc575896a4b0" containerName="cleanup" Oct 30 09:16:33.673824 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.673759 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3bc69d13-9b2a-4544-843b-fc575896a4b0" containerName="cleanup" Oct 30 09:16:33.673824 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.673768 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc69d13-9b2a-4544-843b-fc575896a4b0" containerName="cleanup" Oct 30 09:16:33.674092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.673910 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc69d13-9b2a-4544-843b-fc575896a4b0" containerName="cleanup" Oct 30 09:16:33.674092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.673927 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc698a9a-0198-40f4-aca1-81a195c40b87" containerName="cleanup" Oct 30 09:16:33.674092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.673937 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc69d13-9b2a-4544-843b-fc575896a4b0" containerName="cleanup" Oct 30 09:16:33.674092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.673946 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc698a9a-0198-40f4-aca1-81a195c40b87" containerName="cleanup" Oct 30 09:16:33.674092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.673957 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc69d13-9b2a-4544-843b-fc575896a4b0" containerName="cleanup" Oct 30 09:16:33.677756 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.677727 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.696261 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.696219 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf"] Oct 30 09:16:33.706501 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.706463 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3a61e1\"/\"distinct-w4-3a61e1-kserve-self-signed-certs\"" Oct 30 09:16:33.755319 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.755279 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-home\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.755512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.755335 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tm8t\" (UniqueName: \"kubernetes.io/projected/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-kube-api-access-2tm8t\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.755512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.755475 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-tls-certs\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.755512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.755508 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-tmp-dir\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.755684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.755638 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-model-cache\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.755729 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.755686 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-dshm\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.856856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.856803 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-tls-certs\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.857061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.856867 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-tmp-dir\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.857061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.856943 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-model-cache\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.857061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.857003 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-dshm\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.857221 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.857062 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-home\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.857221 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.857105 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tm8t\" (UniqueName: \"kubernetes.io/projected/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-kube-api-access-2tm8t\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.857485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.857373 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-tmp-dir\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.857485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.857420 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-model-cache\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.857685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.857646 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-home\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.859684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.859658 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-dshm\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.860280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.860255 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-tls-certs\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.877807 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.877764 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tm8t\" (UniqueName: \"kubernetes.io/projected/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-kube-api-access-2tm8t\") pod \"distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:33.996411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:33.996315 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:34.303708 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:34.303681 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf"] Oct 30 09:16:34.306391 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:16:34.306357 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99e9e5d6_3cf4_4d8b_90dc_f7920c8a1ed7.slice/crio-8bbffc02bfd168dd022d8d1ec69da36b6e3db3225e8a25418d0a1a6c81984db2 WatchSource:0}: Error finding container 8bbffc02bfd168dd022d8d1ec69da36b6e3db3225e8a25418d0a1a6c81984db2: Status 404 returned error can't find the container with id 8bbffc02bfd168dd022d8d1ec69da36b6e3db3225e8a25418d0a1a6c81984db2 Oct 30 09:16:34.425163 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:34.425124 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" event={"ID":"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7","Type":"ContainerStarted","Data":"6e43cf073bfcb9320704b0ace76d6bb48bdba976d0420bc5d0604b1a87c39267"} Oct 30 09:16:34.425163 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:34.425172 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" event={"ID":"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7","Type":"ContainerStarted","Data":"8bbffc02bfd168dd022d8d1ec69da36b6e3db3225e8a25418d0a1a6c81984db2"} Oct 30 09:16:34.467220 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:34.467164 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" podStartSLOduration=1.467147671 podStartE2EDuration="1.467147671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:16:34.465647018 +0000 UTC m=+1312.328352840" watchObservedRunningTime="2026-10-30 09:16:34.467147671 +0000 UTC m=+1312.329853496" Oct 30 09:16:34.997034 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:34.996990 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:39.359056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.359018 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr"] Oct 30 09:16:39.359692 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.359672 5938 cpu_manager.go:414] "containerMap: removing container" podUID="dc698a9a-0198-40f4-aca1-81a195c40b87" containerName="cleanup" Oct 30 09:16:39.359793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.359694 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc698a9a-0198-40f4-aca1-81a195c40b87" containerName="cleanup" Oct 30 09:16:39.359889 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.359875 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc698a9a-0198-40f4-aca1-81a195c40b87" containerName="cleanup" Oct 30 09:16:39.363176 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.363153 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.379232 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.379207 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3a61e1-openshift-default-dockercfg-vvdt5\"" Oct 30 09:16:39.394185 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.394150 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr"] Oct 30 09:16:39.522405 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.522357 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.522405 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.522407 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.522696 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.522494 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.522696 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.522554 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.522696 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.522624 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.522696 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.522677 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.522901 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.522736 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s576l\" (UniqueName: \"kubernetes.io/projected/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-kube-api-access-s576l\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.522901 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.522768 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.522901 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.522863 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.623942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.623911 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.624155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.623979 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.624155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.624022 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.624155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.624047 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.624155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.624080 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.624155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.624121 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.624414 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.624164 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.624414 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.624223 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.624414 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.624250 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s576l\" (UniqueName: \"kubernetes.io/projected/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-kube-api-access-s576l\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.624569 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.624511 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.624646 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.624625 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.624785 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.624762 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.624869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.624787 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.625091 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.625067 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.626768 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.626737 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.626895 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.626876 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.645257 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.645223 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.645428 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.645395 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s576l\" (UniqueName: \"kubernetes.io/projected/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-kube-api-access-s576l\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.677844 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.677815 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:39.993663 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.993638 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr"] Oct 30 09:16:39.996775 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:16:39.996735 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13923ad4_0e7f_44f5_8eb3_cd84b0d044c8.slice/crio-c2cc2b01ea135bddd80bd9a7f418dd9385360d26cd5483e3a8d73cebffe27e79 WatchSource:0}: Error finding container c2cc2b01ea135bddd80bd9a7f418dd9385360d26cd5483e3a8d73cebffe27e79: Status 404 returned error can't find the container with id c2cc2b01ea135bddd80bd9a7f418dd9385360d26cd5483e3a8d73cebffe27e79 Oct 30 09:16:39.999393 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.999357 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:16:39.999479 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.999437 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:16:39.999515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:39.999493 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:16:40.461225 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:40.461192 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" event={"ID":"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8","Type":"ContainerStarted","Data":"8e40ec740a8035e020b1fd43aca281322226955dcbb2cf9e3bf125d27c7485e1"} Oct 30 09:16:40.461675 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:40.461247 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" event={"ID":"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8","Type":"ContainerStarted","Data":"c2cc2b01ea135bddd80bd9a7f418dd9385360d26cd5483e3a8d73cebffe27e79"} Oct 30 09:16:40.505915 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:40.505859 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" podStartSLOduration=1.505839934 podStartE2EDuration="1.505839934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:16:40.503871844 +0000 UTC m=+1318.366577663" watchObservedRunningTime="2026-10-30 09:16:40.505839934 +0000 UTC m=+1318.368545755" Oct 30 09:16:40.678149 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:40.678111 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:41.680034 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:41.680005 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 09:16:41.680476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:41.680049 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" podUID="13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 09:16:42.678985 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:42.678943 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 09:16:42.679186 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:42.679007 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" podUID="13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 09:16:43.678797 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:43.678763 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 09:16:43.679349 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:43.678818 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" podUID="13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 09:16:43.997632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:43.996904 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:44.009676 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:44.009648 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:44.497528 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:44.497491 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:16:44.678701 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:44.678665 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 09:16:44.678928 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:44.678738 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" podUID="13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 09:16:45.679115 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:45.679079 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 09:16:45.679544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:45.679149 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" podUID="13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 09:16:45.811769 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:16:45.811733 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:16:46.678982 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:46.678932 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 09:16:46.679262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:46.679011 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" podUID="13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 09:16:48.395795 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:48.395761 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:48.396422 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:48.396109 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:48.397345 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:48.397314 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:16:54.533520 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.533481 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh"] Oct 30 09:16:54.537964 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.537935 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.554134 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.554102 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-176b6c-openshift-default-dockercfg-k67tz\"" Oct 30 09:16:54.570457 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.570415 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh"] Oct 30 09:16:54.707542 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.707496 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.707862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.707547 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpzwd\" (UniqueName: \"kubernetes.io/projected/cda8e58c-2364-4edb-be89-36fc16609681-kube-api-access-fpzwd\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.707862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.707655 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda8e58c-2364-4edb-be89-36fc16609681-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.707862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.707716 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.707862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.707797 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda8e58c-2364-4edb-be89-36fc16609681-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.707862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.707846 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.708139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.707960 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda8e58c-2364-4edb-be89-36fc16609681-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.708139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.708023 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.708139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.708069 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.809435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.809323 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda8e58c-2364-4edb-be89-36fc16609681-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.809435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.809382 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.809757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.809440 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda8e58c-2364-4edb-be89-36fc16609681-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.809757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.809485 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.809757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.809524 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.809757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.809605 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.809757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.809637 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpzwd\" (UniqueName: \"kubernetes.io/projected/cda8e58c-2364-4edb-be89-36fc16609681-kube-api-access-fpzwd\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.809757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.809670 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda8e58c-2364-4edb-be89-36fc16609681-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.809757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.809730 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.810115 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.809976 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.810115 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.810019 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.810398 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.810151 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.810398 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.810253 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.810628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.810575 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda8e58c-2364-4edb-be89-36fc16609681-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.812172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.812134 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.812389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.812370 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda8e58c-2364-4edb-be89-36fc16609681-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.830105 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.830074 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda8e58c-2364-4edb-be89-36fc16609681-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.830268 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.830144 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpzwd\" (UniqueName: \"kubernetes.io/projected/cda8e58c-2364-4edb-be89-36fc16609681-kube-api-access-fpzwd\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:54.853087 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:54.853047 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:55.172907 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:55.172706 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh"] Oct 30 09:16:55.176144 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:16:55.176114 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda8e58c_2364_4edb_be89_36fc16609681.slice/crio-ee15df3fdfe022721aba8ecdf9adb514784fd08099d4d69e01c826c8019a9672 WatchSource:0}: Error finding container ee15df3fdfe022721aba8ecdf9adb514784fd08099d4d69e01c826c8019a9672: Status 404 returned error can't find the container with id ee15df3fdfe022721aba8ecdf9adb514784fd08099d4d69e01c826c8019a9672 Oct 30 09:16:55.178878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:55.178834 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:16:55.179022 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:55.178942 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:16:55.179022 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:55.178975 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:16:55.552938 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:55.552833 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" event={"ID":"cda8e58c-2364-4edb-be89-36fc16609681","Type":"ContainerStarted","Data":"243f303b17815d116e7f0eac32302a4befd813562dd9217ca1def3fe16ebb129"} Oct 30 09:16:55.552938 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:55.552890 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" event={"ID":"cda8e58c-2364-4edb-be89-36fc16609681","Type":"ContainerStarted","Data":"ee15df3fdfe022721aba8ecdf9adb514784fd08099d4d69e01c826c8019a9672"} Oct 30 09:16:55.596650 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:55.596570 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" podStartSLOduration=1.5965494439999999 podStartE2EDuration="1.596549444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:16:55.595468669 +0000 UTC m=+1333.458174490" watchObservedRunningTime="2026-10-30 09:16:55.596549444 +0000 UTC m=+1333.459255266" Oct 30 09:16:55.853516 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:55.853426 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:16:56.854569 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:56.854530 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 09:16:56.855061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:56.854628 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" podUID="cda8e58c-2364-4edb-be89-36fc16609681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 09:16:57.853870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:57.853827 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 09:16:57.854065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:57.853888 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" podUID="cda8e58c-2364-4edb-be89-36fc16609681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 09:16:58.854494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:58.853862 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 09:16:58.854494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:58.853922 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" podUID="cda8e58c-2364-4edb-be89-36fc16609681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 09:16:59.853621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:59.853548 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 09:16:59.853919 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:16:59.853654 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" podUID="cda8e58c-2364-4edb-be89-36fc16609681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 09:17:00.854192 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:00.854156 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 09:17:00.854664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:00.854217 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" podUID="cda8e58c-2364-4edb-be89-36fc16609681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 09:17:01.854322 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:01.854288 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 09:17:01.854762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:01.854363 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" podUID="cda8e58c-2364-4edb-be89-36fc16609681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 09:17:02.859014 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:02.858984 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:17:02.859524 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:02.859290 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:17:02.860115 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:02.860095 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:17:15.810425 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:17:15.810386 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:17:26.148972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.148882 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8"] Oct 30 09:17:26.152934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.152896 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.173017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.172987 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8"] Oct 30 09:17:26.184151 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.184094 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e3d2ff\"/\"distinct-w2-e3d2ff-kserve-self-signed-certs\"" Oct 30 09:17:26.188668 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.188614 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-tmp-dir\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.188668 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.188679 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-model-cache\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.188971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.188804 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71e1eff7-c792-4bee-a800-68580ce486f3-tls-certs\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.189033 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.188969 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtnxg\" (UniqueName: \"kubernetes.io/projected/71e1eff7-c792-4bee-a800-68580ce486f3-kube-api-access-xtnxg\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.189033 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.188994 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-dshm\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.189033 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.189013 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-home\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.289874 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.289830 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtnxg\" (UniqueName: \"kubernetes.io/projected/71e1eff7-c792-4bee-a800-68580ce486f3-kube-api-access-xtnxg\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.290077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.289890 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-dshm\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.290077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.289924 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-home\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.290077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.289995 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-tmp-dir\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.290259 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.290127 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-model-cache\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.290259 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.290212 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71e1eff7-c792-4bee-a800-68580ce486f3-tls-certs\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.290462 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.290436 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-tmp-dir\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.290654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.290630 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-home\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.290833 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.290543 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-model-cache\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.293065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.293021 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-dshm\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.293241 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.293215 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71e1eff7-c792-4bee-a800-68580ce486f3-tls-certs\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.310901 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.310870 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtnxg\" (UniqueName: \"kubernetes.io/projected/71e1eff7-c792-4bee-a800-68580ce486f3-kube-api-access-xtnxg\") pod \"distinct-w2-e3d2ff-kserve-7bb95f665-287z8\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.469302 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.469183 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:26.772816 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:26.772785 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8"] Oct 30 09:17:26.775365 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:17:26.775336 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71e1eff7_c792_4bee_a800_68580ce486f3.slice/crio-5220ee0b504b044c6eda6ab20cfc243968f70a59fb5feeabac758b90a981eb9f WatchSource:0}: Error finding container 5220ee0b504b044c6eda6ab20cfc243968f70a59fb5feeabac758b90a981eb9f: Status 404 returned error can't find the container with id 5220ee0b504b044c6eda6ab20cfc243968f70a59fb5feeabac758b90a981eb9f Oct 30 09:17:27.756685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:27.756639 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" event={"ID":"71e1eff7-c792-4bee-a800-68580ce486f3","Type":"ContainerStarted","Data":"2ad503550e79033bd6d23aef40fbb4df33dd9d6b26d7d74bedcc9589f06d351f"} Oct 30 09:17:27.757103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:27.756698 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" event={"ID":"71e1eff7-c792-4bee-a800-68580ce486f3","Type":"ContainerStarted","Data":"5220ee0b504b044c6eda6ab20cfc243968f70a59fb5feeabac758b90a981eb9f"} Oct 30 09:17:27.800086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:27.800025 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" podStartSLOduration=1.800006894 podStartE2EDuration="1.800006894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:17:27.797757478 +0000 UTC m=+1365.660463298" watchObservedRunningTime="2026-10-30 09:17:27.800006894 +0000 UTC m=+1365.662712715" Oct 30 09:17:28.470256 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:28.470212 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:36.469685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:36.469651 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:36.482528 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:36.482498 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:36.821779 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:36.821680 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:17:45.812044 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:17:45.812001 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:17:50.417359 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.417325 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb"] Oct 30 09:17:50.422122 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.422098 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.438161 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.438132 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2f69c8-openshift-default-dockercfg-hmrnb\"" Oct 30 09:17:50.454341 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.454300 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb"] Oct 30 09:17:50.467527 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.467475 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wg7f\" (UniqueName: \"kubernetes.io/projected/6e0e5389-b8e8-4033-a7b3-98830bf147af-kube-api-access-6wg7f\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.467527 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.467517 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e0e5389-b8e8-4033-a7b3-98830bf147af-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.467844 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.467626 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.467844 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.467738 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.467844 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.467768 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.467844 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.467845 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.468072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.467909 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.468072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.468011 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.468072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.468032 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.568910 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.568865 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.568910 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.568911 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.569244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.568946 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wg7f\" (UniqueName: \"kubernetes.io/projected/6e0e5389-b8e8-4033-a7b3-98830bf147af-kube-api-access-6wg7f\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.569244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.568969 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e0e5389-b8e8-4033-a7b3-98830bf147af-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.569244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.569013 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.569244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.569105 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.569244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.569146 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.569244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.569185 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.569544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.569277 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.569544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.569342 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.569544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.569353 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.569544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.569501 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.569787 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.569633 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.569787 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.569741 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e0e5389-b8e8-4033-a7b3-98830bf147af-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.572016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.571983 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.572150 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.572039 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.589771 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.589738 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.589946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.589921 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wg7f\" (UniqueName: \"kubernetes.io/projected/6e0e5389-b8e8-4033-a7b3-98830bf147af-kube-api-access-6wg7f\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:50.736811 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:50.736711 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:51.069239 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:51.069171 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb"] Oct 30 09:17:51.071924 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:17:51.071892 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0e5389_b8e8_4033_a7b3_98830bf147af.slice/crio-fbd35e038a25badc123253bbd13339d27de54c5113d40f256a984ff4cedb977b WatchSource:0}: Error finding container fbd35e038a25badc123253bbd13339d27de54c5113d40f256a984ff4cedb977b: Status 404 returned error can't find the container with id fbd35e038a25badc123253bbd13339d27de54c5113d40f256a984ff4cedb977b Oct 30 09:17:51.074843 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:51.074791 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:17:51.075027 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:51.074983 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:17:51.075117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:51.075089 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:17:51.901817 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:51.901768 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" event={"ID":"6e0e5389-b8e8-4033-a7b3-98830bf147af","Type":"ContainerStarted","Data":"a96e59da7fa89e0012a02cd670ab39b4f32a1db680a18408bfaeee06308630bc"} Oct 30 09:17:51.902351 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:51.901844 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" event={"ID":"6e0e5389-b8e8-4033-a7b3-98830bf147af","Type":"ContainerStarted","Data":"fbd35e038a25badc123253bbd13339d27de54c5113d40f256a984ff4cedb977b"} Oct 30 09:17:51.987212 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:51.987130 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" podStartSLOduration=1.9871112210000001 podStartE2EDuration="1.987111221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:17:51.961902956 +0000 UTC m=+1389.824608804" watchObservedRunningTime="2026-10-30 09:17:51.987111221 +0000 UTC m=+1389.849817047" Oct 30 09:17:52.739619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:52.739012 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:52.748617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:52.742255 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 09:17:52.748617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:52.742320 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" podUID="6e0e5389-b8e8-4033-a7b3-98830bf147af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 09:17:53.741613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:53.740988 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 09:17:53.741613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:53.741050 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" podUID="6e0e5389-b8e8-4033-a7b3-98830bf147af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 09:17:54.741615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:54.740685 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 09:17:54.741615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:54.740757 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" podUID="6e0e5389-b8e8-4033-a7b3-98830bf147af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 09:17:55.738096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:55.738052 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 09:17:55.738427 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:55.738395 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" podUID="6e0e5389-b8e8-4033-a7b3-98830bf147af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 09:17:56.742102 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:56.742060 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 09:17:56.742814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:56.742776 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" podUID="6e0e5389-b8e8-4033-a7b3-98830bf147af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 09:17:57.737116 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:57.737076 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 09:17:57.737314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:57.737148 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" podUID="6e0e5389-b8e8-4033-a7b3-98830bf147af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 09:17:58.742551 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:58.742515 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:58.743045 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:58.742861 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:17:58.743875 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:17:58.743856 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:18:02.483966 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.483916 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc"] Oct 30 09:18:02.487807 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.487774 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.504951 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.504921 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c5447d-openshift-default-dockercfg-nx55s\"" Oct 30 09:18:02.521077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.521043 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc"] Oct 30 09:18:02.537247 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.537215 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.537247 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.537249 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.537494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.537289 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.537494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.537464 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfzmp\" (UniqueName: \"kubernetes.io/projected/00c9d38a-98f0-4598-8650-d49c9f16be07-kube-api-access-cfzmp\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.537577 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.537521 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c9d38a-98f0-4598-8650-d49c9f16be07-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.537658 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.537638 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.537702 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.537686 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.537801 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.537786 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.537859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.537843 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.639044 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.639005 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.639044 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.639042 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.639308 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.639129 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.639308 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.639152 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.639308 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.639181 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.639469 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.639330 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzmp\" (UniqueName: \"kubernetes.io/projected/00c9d38a-98f0-4598-8650-d49c9f16be07-kube-api-access-cfzmp\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.639469 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.639404 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c9d38a-98f0-4598-8650-d49c9f16be07-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.639469 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.639454 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.639664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.639469 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.639664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.639487 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.639664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.639527 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.639816 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.639782 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.640133 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.640107 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.640237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.640200 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c9d38a-98f0-4598-8650-d49c9f16be07-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.642103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.642079 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.642210 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.642111 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.661129 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.661095 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.661393 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.661368 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfzmp\" (UniqueName: \"kubernetes.io/projected/00c9d38a-98f0-4598-8650-d49c9f16be07-kube-api-access-cfzmp\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:02.803288 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:02.803181 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:03.131805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:03.131774 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc"] Oct 30 09:18:03.135148 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:18:03.135119 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c9d38a_98f0_4598_8650_d49c9f16be07.slice/crio-36d2d5dc86183a921f6d917f44cd71bd95bf31a42941b315c6d5d3265b8887fa WatchSource:0}: Error finding container 36d2d5dc86183a921f6d917f44cd71bd95bf31a42941b315c6d5d3265b8887fa: Status 404 returned error can't find the container with id 36d2d5dc86183a921f6d917f44cd71bd95bf31a42941b315c6d5d3265b8887fa Oct 30 09:18:03.138454 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:03.138417 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:03.138566 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:03.138496 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:03.138566 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:03.138525 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:03.994474 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:03.994434 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" event={"ID":"00c9d38a-98f0-4598-8650-d49c9f16be07","Type":"ContainerStarted","Data":"06753652c39f2159a69e910c5dd6fdf02c976f54b7f8325680e2cb8c129db0a8"} Oct 30 09:18:03.994990 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:03.994507 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" event={"ID":"00c9d38a-98f0-4598-8650-d49c9f16be07","Type":"ContainerStarted","Data":"36d2d5dc86183a921f6d917f44cd71bd95bf31a42941b315c6d5d3265b8887fa"} Oct 30 09:18:04.070620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:04.055227 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" podStartSLOduration=2.055200519 podStartE2EDuration="2.055200519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:04.054125338 +0000 UTC m=+1401.916831161" watchObservedRunningTime="2026-10-30 09:18:04.055200519 +0000 UTC m=+1401.917906341" Oct 30 09:18:04.803854 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:04.803800 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:04.805658 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:04.805633 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 09:18:04.805759 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:04.805687 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" podUID="00c9d38a-98f0-4598-8650-d49c9f16be07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 09:18:05.804427 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:05.804393 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 09:18:05.804878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:05.804453 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" podUID="00c9d38a-98f0-4598-8650-d49c9f16be07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 09:18:06.803769 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:06.803729 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 09:18:06.803985 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:06.803784 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" podUID="00c9d38a-98f0-4598-8650-d49c9f16be07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 09:18:07.803660 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:07.803619 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 09:18:07.804110 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:07.803689 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" podUID="00c9d38a-98f0-4598-8650-d49c9f16be07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 09:18:08.804598 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:08.804532 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 09:18:08.805027 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:08.804628 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" podUID="00c9d38a-98f0-4598-8650-d49c9f16be07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 09:18:09.805127 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:09.805084 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 09:18:09.828615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:09.818196 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" podUID="00c9d38a-98f0-4598-8650-d49c9f16be07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 09:18:10.804698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:10.804655 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 09:18:10.804927 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:10.804725 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" podUID="00c9d38a-98f0-4598-8650-d49c9f16be07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 09:18:12.669219 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:12.669186 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:12.669759 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:12.669618 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:12.670749 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:12.670727 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:18:15.813138 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:18:15.813094 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:18:24.709256 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.709213 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx"] Oct 30 09:18:24.713688 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.713657 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.762511 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.762480 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx"] Oct 30 09:18:24.878408 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.878367 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.878667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.878449 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.878667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.878516 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5dtb\" (UniqueName: \"kubernetes.io/projected/a464a5b4-c19b-4e64-8b9b-7b8459471501-kube-api-access-k5dtb\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.878667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.878649 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.878814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.878759 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.878814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.878777 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.878886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.878829 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a464a5b4-c19b-4e64-8b9b-7b8459471501-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.878886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.878870 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.878955 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.878888 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.979920 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.979811 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.979920 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.979866 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5dtb\" (UniqueName: \"kubernetes.io/projected/a464a5b4-c19b-4e64-8b9b-7b8459471501-kube-api-access-k5dtb\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.979920 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.979915 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.980204 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.980019 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.980204 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.980043 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.980204 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.980096 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a464a5b4-c19b-4e64-8b9b-7b8459471501-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.980204 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.980135 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.980204 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.980165 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.980479 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.980235 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.980479 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.980304 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.980479 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.980409 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.980615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.980519 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.980669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.980580 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.981252 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.981195 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a464a5b4-c19b-4e64-8b9b-7b8459471501-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.982813 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.982785 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:24.983137 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:24.983115 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:25.004462 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:25.004428 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:25.004827 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:25.004802 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5dtb\" (UniqueName: \"kubernetes.io/projected/a464a5b4-c19b-4e64-8b9b-7b8459471501-kube-api-access-k5dtb\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:25.031097 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:25.031058 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:25.368166 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:25.368131 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx"] Oct 30 09:18:25.371119 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:18:25.371083 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda464a5b4_c19b_4e64_8b9b_7b8459471501.slice/crio-953229ad21718d0cf8ca822730cc3a440a9c2b4e11c1975e02af9e3d32d75679 WatchSource:0}: Error finding container 953229ad21718d0cf8ca822730cc3a440a9c2b4e11c1975e02af9e3d32d75679: Status 404 returned error can't find the container with id 953229ad21718d0cf8ca822730cc3a440a9c2b4e11c1975e02af9e3d32d75679 Oct 30 09:18:25.373958 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:25.373916 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:25.374040 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:25.373998 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:25.374040 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:25.374027 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:26.165531 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:26.165489 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" event={"ID":"a464a5b4-c19b-4e64-8b9b-7b8459471501","Type":"ContainerStarted","Data":"07223fadf24fdcf584ad275072c2ce4e9017cfecb07d6368c0092f416a11e930"} Oct 30 09:18:26.166013 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:26.165567 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" event={"ID":"a464a5b4-c19b-4e64-8b9b-7b8459471501","Type":"ContainerStarted","Data":"953229ad21718d0cf8ca822730cc3a440a9c2b4e11c1975e02af9e3d32d75679"} Oct 30 09:18:27.045056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:27.045016 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:27.064127 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:27.064067 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 09:18:27.064444 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:27.064406 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" podUID="a464a5b4-c19b-4e64-8b9b-7b8459471501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 09:18:28.040380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:28.040337 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 09:18:28.043886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:28.043810 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" podUID="a464a5b4-c19b-4e64-8b9b-7b8459471501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 09:18:29.035167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:29.035124 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 09:18:29.035440 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:29.035405 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" podUID="a464a5b4-c19b-4e64-8b9b-7b8459471501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 09:18:30.042717 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:30.033081 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 09:18:30.042717 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:30.033169 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" podUID="a464a5b4-c19b-4e64-8b9b-7b8459471501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 09:18:31.038619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:31.034041 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 09:18:31.038619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:31.034127 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" podUID="a464a5b4-c19b-4e64-8b9b-7b8459471501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 09:18:32.031847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.031800 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 09:18:32.032369 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.031889 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" podUID="a464a5b4-c19b-4e64-8b9b-7b8459471501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 09:18:32.631495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.631432 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" podStartSLOduration=8.631415641 podStartE2EDuration="8.631415641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:26.248704186 +0000 UTC m=+1424.111410008" watchObservedRunningTime="2026-10-30 09:18:32.631415641 +0000 UTC m=+1430.494121465" Oct 30 09:18:32.633180 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.633148 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz"] Oct 30 09:18:32.637091 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.637059 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.683642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.683347 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz"] Oct 30 09:18:32.721566 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.721510 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.721790 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.721645 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.721790 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.721763 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.721907 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.721834 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.721964 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.721918 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.722065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.722046 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.722127 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.722114 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.722181 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.722152 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7657r\" (UniqueName: \"kubernetes.io/projected/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-kube-api-access-7657r\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.722236 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.722201 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.823138 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.823089 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.823138 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.823142 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.823433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.823253 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.823433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.823293 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.823433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.823326 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7657r\" (UniqueName: \"kubernetes.io/projected/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-kube-api-access-7657r\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.823433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.823374 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.823757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.823462 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.823757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.823548 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.823757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.823650 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.823757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.823722 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.823994 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.823968 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.824176 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.824020 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.824313 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.824277 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.824401 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.824166 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.826700 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.826669 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.826827 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.826779 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.846899 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.846866 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.847228 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.847203 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7657r\" (UniqueName: \"kubernetes.io/projected/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-kube-api-access-7657r\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:32.957401 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:32.957296 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:33.124742 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:33.124713 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:33.125253 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:33.125134 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:33.126877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:33.126840 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:18:33.329562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:33.329524 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz"] Oct 30 09:18:33.332049 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:18:33.332019 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6de3fa7_ddd5_4816_b6b9_f18bae20bf0d.slice/crio-3bd6aa06018ce7362e2d3a6e96bc616e70f50ca5fb594b94383d316f5bb74d26 WatchSource:0}: Error finding container 3bd6aa06018ce7362e2d3a6e96bc616e70f50ca5fb594b94383d316f5bb74d26: Status 404 returned error can't find the container with id 3bd6aa06018ce7362e2d3a6e96bc616e70f50ca5fb594b94383d316f5bb74d26 Oct 30 09:18:33.334672 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:33.334630 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:33.334754 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:33.334715 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:33.334791 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:33.334755 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:34.248191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:34.248151 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" event={"ID":"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d","Type":"ContainerStarted","Data":"f0b6839f6c78f341a062fba66fc4a213d5a723b7ddf814b28ccf5ed696aabf41"} Oct 30 09:18:34.248813 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:34.248781 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" event={"ID":"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d","Type":"ContainerStarted","Data":"3bd6aa06018ce7362e2d3a6e96bc616e70f50ca5fb594b94383d316f5bb74d26"} Oct 30 09:18:34.305178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:34.305123 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" podStartSLOduration=2.305106344 podStartE2EDuration="2.305106344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:34.30355467 +0000 UTC m=+1432.166260492" watchObservedRunningTime="2026-10-30 09:18:34.305106344 +0000 UTC m=+1432.167812165" Oct 30 09:18:34.958379 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:34.958345 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:34.960174 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:34.960144 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:18:34.960313 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:34.960208 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" podUID="e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 09:18:35.603536 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.603495 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6"] Oct 30 09:18:35.607405 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.607380 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.661755 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.661700 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.661976 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.661847 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.661976 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.661881 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.661976 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.661937 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fddb9f28-1159-49f5-8982-7e0c8275df4a-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.662165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.661984 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.662165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.662103 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.662300 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.662276 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.662517 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.662495 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.662651 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.662536 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thrtx\" (UniqueName: \"kubernetes.io/projected/fddb9f28-1159-49f5-8982-7e0c8275df4a-kube-api-access-thrtx\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.668701 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.668668 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6"] Oct 30 09:18:35.764271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.764226 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.764271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.764269 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.764510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.764320 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fddb9f28-1159-49f5-8982-7e0c8275df4a-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.764510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.764365 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.764510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.764442 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.764649 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.764524 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.764649 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.764552 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.764649 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.764607 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thrtx\" (UniqueName: \"kubernetes.io/projected/fddb9f28-1159-49f5-8982-7e0c8275df4a-kube-api-access-thrtx\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.764649 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.764636 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.764801 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.764754 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.765100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.765071 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.765251 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.765229 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.765526 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.765500 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.765964 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.765943 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fddb9f28-1159-49f5-8982-7e0c8275df4a-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.767345 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.767322 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.768208 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.768188 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.788538 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.788503 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.788971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.788940 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thrtx\" (UniqueName: \"kubernetes.io/projected/fddb9f28-1159-49f5-8982-7e0c8275df4a-kube-api-access-thrtx\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:35.958135 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.958083 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:18:35.958366 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.958175 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" podUID="e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 09:18:35.962218 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:35.962184 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:36.346512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:36.346482 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6"] Oct 30 09:18:36.349344 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:18:36.349302 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddb9f28_1159_49f5_8982_7e0c8275df4a.slice/crio-851f1b74e6258a661fc266ec9a4593f56262c04c2e3341123ce54806bf298921 WatchSource:0}: Error finding container 851f1b74e6258a661fc266ec9a4593f56262c04c2e3341123ce54806bf298921: Status 404 returned error can't find the container with id 851f1b74e6258a661fc266ec9a4593f56262c04c2e3341123ce54806bf298921 Oct 30 09:18:36.351950 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:36.351909 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:36.352056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:36.351987 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:36.352056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:36.352016 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:36.958758 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:36.958719 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:18:36.959259 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:36.958791 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" podUID="e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 09:18:37.278012 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:37.277921 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" event={"ID":"fddb9f28-1159-49f5-8982-7e0c8275df4a","Type":"ContainerStarted","Data":"20087ed558a52d49cc2cdaa63fa58637332462035248de52de79c6d3a9c934bb"} Oct 30 09:18:37.278012 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:37.277973 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" event={"ID":"fddb9f28-1159-49f5-8982-7e0c8275df4a","Type":"ContainerStarted","Data":"851f1b74e6258a661fc266ec9a4593f56262c04c2e3341123ce54806bf298921"} Oct 30 09:18:37.369920 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:37.369059 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" podStartSLOduration=2.3690332720000002 podStartE2EDuration="2.369033272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:37.343949274 +0000 UTC m=+1435.206655096" watchObservedRunningTime="2026-10-30 09:18:37.369033272 +0000 UTC m=+1435.231739094" Oct 30 09:18:37.991061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:37.989660 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:38.036814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:38.036777 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:18:38.037148 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:38.037113 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" podUID="e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 09:18:38.054115 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:38.054063 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 09:18:38.054632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:38.054581 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" podUID="fddb9f28-1159-49f5-8982-7e0c8275df4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 09:18:38.769897 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:38.769835 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:38.930620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:38.929658 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:38.953719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:38.952998 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 09:18:38.960156 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:38.960116 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:18:38.960504 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:38.960469 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" podUID="e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 09:18:38.985835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:38.985792 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 09:18:38.986155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:38.986115 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" podUID="fddb9f28-1159-49f5-8982-7e0c8275df4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 09:18:39.021806 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:39.021692 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 09:18:39.043662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:39.042707 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:39.064615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:39.061023 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:39.666627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:39.665355 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:39.786878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:39.771715 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 09:18:39.786878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:39.772496 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:18:39.813765 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:39.813704 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:40.042297 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.042181 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:18:40.063977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.063943 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" podUID="e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 09:18:40.064244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.043518 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 09:18:40.090573 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.043532 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 09:18:40.091331 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.043164 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:40.137620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.093731 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:18:40.137620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.094921 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:18:40.137620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.094964 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:18:40.137620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.063717 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:40.137620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.043010 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 09:18:40.137620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.096961 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" podUID="fddb9f28-1159-49f5-8982-7e0c8275df4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 09:18:40.138283 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.138247 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:18:40.185618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.177943 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4"] Oct 30 09:18:40.468016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.467976 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4"] Oct 30 09:18:40.526616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.488316 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.607697 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.607664 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.608067 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.608048 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fce2865-c606-45f4-84d0-6691b9768fc7-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.608257 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.608240 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhrpd\" (UniqueName: \"kubernetes.io/projected/4fce2865-c606-45f4-84d0-6691b9768fc7-kube-api-access-fhrpd\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.608382 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.608369 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.608668 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.608652 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.608826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.608812 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.608978 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.608964 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.609108 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.609095 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.609311 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.492740 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.620999 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.492790 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" podUID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:40.683097 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.683042 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:40.695700 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.695665 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:18:40.799978 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.799849 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:40.809617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.809541 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.811840 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.811741 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.811840 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.811801 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.811840 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.811841 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.812149 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.811942 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.812149 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.811984 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.812149 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.812041 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fce2865-c606-45f4-84d0-6691b9768fc7-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.812149 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.812116 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhrpd\" (UniqueName: \"kubernetes.io/projected/4fce2865-c606-45f4-84d0-6691b9768fc7-kube-api-access-fhrpd\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.812149 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.812145 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.816893 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.816863 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fce2865-c606-45f4-84d0-6691b9768fc7-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.831004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.830958 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.835808 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.835761 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.845564 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.845519 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.845876 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.845852 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:40.865178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.865146 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.908616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.886209 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.929037 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.928999 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:40.984960 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.983687 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:40.986982 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.985271 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 09:18:40.987286 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.987260 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" podUID="fddb9f28-1159-49f5-8982-7e0c8275df4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 09:18:40.987426 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.986300 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:18:40.987543 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:40.987525 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" podUID="e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 09:18:41.012647 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.010636 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhrpd\" (UniqueName: \"kubernetes.io/projected/4fce2865-c606-45f4-84d0-6691b9768fc7-kube-api-access-fhrpd\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:41.071724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.069737 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:41.102761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.095678 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:41.102761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.095723 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:41.102761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.096126 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:41.102761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.096675 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:41.102761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.096787 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:18:41.145873 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.145834 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:41.182448 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.182405 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:18:41.268618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.266300 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:18:41.500420 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.497298 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:41.683695 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.674836 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" podUID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 09:18:41.683695 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.675560 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:18:41.906617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.880160 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4"] Oct 30 09:18:41.963098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.963058 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:18:41.963389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.963360 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" podUID="e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 09:18:42.002619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.972275 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:18:42.002619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.974994 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 09:18:42.002619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.975047 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" podUID="fddb9f28-1159-49f5-8982-7e0c8275df4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 09:18:42.002619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:41.994122 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:18:42.048012 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:42.040804 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:42.048012 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:42.040939 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:42.048012 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:42.041019 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:42.109389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:42.109341 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:42.110413 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:42.110370 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:42.253483 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:42.163984 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:18:42.284726 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:42.280250 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:18:42.527667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:42.527050 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" event={"ID":"4fce2865-c606-45f4-84d0-6691b9768fc7","Type":"ContainerStarted","Data":"e8cdf4906214880aaabfc38e9a3691a0ac56a57cf4e55858f373b39b6af6f15f"} Oct 30 09:18:42.527667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:42.527153 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" event={"ID":"4fce2865-c606-45f4-84d0-6691b9768fc7","Type":"ContainerStarted","Data":"60a9203e12c325cf30283013bdb4c7c2841e0776e880dbe9a38ee6f127bf19f4"} Oct 30 09:18:42.603714 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:42.603636 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" podStartSLOduration=3.603614636 podStartE2EDuration="3.603614636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:42.601751924 +0000 UTC m=+1440.464457960" watchObservedRunningTime="2026-10-30 09:18:42.603614636 +0000 UTC m=+1440.466320463" Oct 30 09:18:42.967318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:42.967279 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 09:18:42.967654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:42.967622 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" podUID="fddb9f28-1159-49f5-8982-7e0c8275df4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 09:18:43.153615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:43.152626 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 09:18:43.153615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:43.152655 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:43.153615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:43.152695 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" podUID="4fce2865-c606-45f4-84d0-6691b9768fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 09:18:43.969615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:43.968717 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:18:43.969615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:43.968792 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" podUID="e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:43.970620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:43.969921 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 09:18:43.970620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:43.969976 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" podUID="fddb9f28-1159-49f5-8982-7e0c8275df4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 09:18:44.159117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:44.159077 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 09:18:44.161104 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:44.159725 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" podUID="4fce2865-c606-45f4-84d0-6691b9768fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 09:18:44.974056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:44.974008 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:44.977658 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:44.975949 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:44.978634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:44.977060 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 09:18:44.979993 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:44.979865 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:18:44.980755 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:44.979943 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" podUID="fddb9f28-1159-49f5-8982-7e0c8275df4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 09:18:45.152701 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:45.147891 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 09:18:45.152701 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:45.147954 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" podUID="4fce2865-c606-45f4-84d0-6691b9768fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 09:18:45.815324 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:18:45.815146 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:18:46.158749 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:46.154351 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 09:18:46.158749 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:46.158185 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" podUID="4fce2865-c606-45f4-84d0-6691b9768fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 09:18:46.610993 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:46.610888 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:46.611339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:46.611315 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:46.612967 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:46.612940 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 09:18:47.150179 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.150139 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 09:18:47.150686 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.150225 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" podUID="4fce2865-c606-45f4-84d0-6691b9768fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 09:18:47.699909 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.699863 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm"] Oct 30 09:18:47.705528 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.705498 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.747434 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.747388 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm"] Oct 30 09:18:47.790292 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.790252 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.790514 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.790310 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.790514 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.790381 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.790514 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.790419 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.790514 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.790451 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.790514 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.790510 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvtr6\" (UniqueName: \"kubernetes.io/projected/56c07cf9-ff35-40b1-96f3-e0188c19e626-kube-api-access-hvtr6\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.790808 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.790552 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.793806 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.793775 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.794006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.793991 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c07cf9-ff35-40b1-96f3-e0188c19e626-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.895308 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.895254 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.895308 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.895312 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.895541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.895366 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.895541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.895394 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.895541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.895416 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.895541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.895455 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvtr6\" (UniqueName: \"kubernetes.io/projected/56c07cf9-ff35-40b1-96f3-e0188c19e626-kube-api-access-hvtr6\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.895541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.895482 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.895541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.895518 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.895541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.895539 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c07cf9-ff35-40b1-96f3-e0188c19e626-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.896366 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.896342 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c07cf9-ff35-40b1-96f3-e0188c19e626-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.896807 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.896638 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.897424 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.897335 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.897424 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.897361 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.898347 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.898323 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.899101 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.899018 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.901315 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.901252 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.963690 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.960221 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:47.963690 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:47.963680 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvtr6\" (UniqueName: \"kubernetes.io/projected/56c07cf9-ff35-40b1-96f3-e0188c19e626-kube-api-access-hvtr6\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:48.056617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:48.026891 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:48.157128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:48.157089 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 09:18:48.157743 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:48.157710 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" podUID="4fce2865-c606-45f4-84d0-6691b9768fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 09:18:48.461134 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:48.461096 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm"] Oct 30 09:18:48.462646 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:18:48.462618 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c07cf9_ff35_40b1_96f3_e0188c19e626.slice/crio-de63f31450b480e7e5c834240383305b3860f0f037e70a45e60ad6a35b218e69 WatchSource:0}: Error finding container de63f31450b480e7e5c834240383305b3860f0f037e70a45e60ad6a35b218e69: Status 404 returned error can't find the container with id de63f31450b480e7e5c834240383305b3860f0f037e70a45e60ad6a35b218e69 Oct 30 09:18:48.465298 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:48.465254 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:48.465408 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:48.465334 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:48.465408 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:48.465372 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:48.647617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:48.638191 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" event={"ID":"56c07cf9-ff35-40b1-96f3-e0188c19e626","Type":"ContainerStarted","Data":"9a16d5c3793f14dd09569b0459516aee1c30bce9baf6d1c846946e24172dbe73"} Oct 30 09:18:48.647617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:48.638254 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" event={"ID":"56c07cf9-ff35-40b1-96f3-e0188c19e626","Type":"ContainerStarted","Data":"de63f31450b480e7e5c834240383305b3860f0f037e70a45e60ad6a35b218e69"} Oct 30 09:18:48.696517 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:48.696467 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" podStartSLOduration=1.696448952 podStartE2EDuration="1.696448952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:48.695932496 +0000 UTC m=+1446.558638318" watchObservedRunningTime="2026-10-30 09:18:48.696448952 +0000 UTC m=+1446.559154773" Oct 30 09:18:49.027681 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.027638 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:18:49.029473 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.029440 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 09:18:49.029698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.029498 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 09:18:49.148011 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.147973 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 09:18:49.148199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.148042 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" podUID="4fce2865-c606-45f4-84d0-6691b9768fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 09:18:49.236635 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.236578 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm"] Oct 30 09:18:49.240861 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.240839 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.263878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.263847 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e3d2ff-openshift-default-dockercfg-j9w6q\"" Oct 30 09:18:49.274792 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.274756 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm"] Oct 30 09:18:49.317532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.317427 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.317532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.317490 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.317836 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.317607 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg7nw\" (UniqueName: \"kubernetes.io/projected/e40f5ce9-777c-44d7-8325-6bb672235db7-kube-api-access-mg7nw\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.318823 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.318791 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.319000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.318920 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.319093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.319037 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.319169 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.319095 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.319169 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.319122 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.319287 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.319210 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40f5ce9-777c-44d7-8325-6bb672235db7-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.423526 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.423459 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.424398 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.424368 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.424687 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.424662 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg7nw\" (UniqueName: \"kubernetes.io/projected/e40f5ce9-777c-44d7-8325-6bb672235db7-kube-api-access-mg7nw\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.424859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.424839 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.425030 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.425013 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.425228 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.425211 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.425387 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.425371 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.425503 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.425488 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.425678 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.425658 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40f5ce9-777c-44d7-8325-6bb672235db7-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.426189 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.426153 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.426293 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.424260 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.426494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.426469 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.426991 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.426968 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40f5ce9-777c-44d7-8325-6bb672235db7-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.427112 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.427085 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.451617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.450663 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.451617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.450939 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.453515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.453483 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.453992 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.453963 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg7nw\" (UniqueName: \"kubernetes.io/projected/e40f5ce9-777c-44d7-8325-6bb672235db7-kube-api-access-mg7nw\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.558060 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.558016 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:49.919028 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.918919 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm"] Oct 30 09:18:49.948614 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:18:49.943797 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40f5ce9_777c_44d7_8325_6bb672235db7.slice/crio-5fba1bb871e635f0ae4e2b70119325d7a5fba06f980fe65395087006afeec42f WatchSource:0}: Error finding container 5fba1bb871e635f0ae4e2b70119325d7a5fba06f980fe65395087006afeec42f: Status 404 returned error can't find the container with id 5fba1bb871e635f0ae4e2b70119325d7a5fba06f980fe65395087006afeec42f Oct 30 09:18:49.948983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.948725 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:49.948983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.948816 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:49.948983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:49.948859 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:50.042153 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:50.042102 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 09:18:50.042378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:50.042177 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 09:18:50.392756 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:50.392720 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:50.393289 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:50.393263 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:50.394475 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:50.394452 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:18:50.662172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:50.662055 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" event={"ID":"e40f5ce9-777c-44d7-8325-6bb672235db7","Type":"ContainerStarted","Data":"232d928ec1daf5303c9f5aa0908a2bad62550ae6110b7ae0275d0bfbc071fd99"} Oct 30 09:18:50.662172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:50.662139 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" event={"ID":"e40f5ce9-777c-44d7-8325-6bb672235db7","Type":"ContainerStarted","Data":"5fba1bb871e635f0ae4e2b70119325d7a5fba06f980fe65395087006afeec42f"} Oct 30 09:18:50.744883 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:50.744813 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" podStartSLOduration=1.7447940640000001 podStartE2EDuration="1.744794064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:50.744040746 +0000 UTC m=+1448.606746571" watchObservedRunningTime="2026-10-30 09:18:50.744794064 +0000 UTC m=+1448.607499884" Oct 30 09:18:51.028201 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:51.028097 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 09:18:51.028476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:51.028449 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 09:18:51.561702 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:51.561660 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:18:51.562768 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:51.562736 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 09:18:51.563030 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:51.563001 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" podUID="e40f5ce9-777c-44d7-8325-6bb672235db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 09:18:52.042326 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:52.042285 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 09:18:52.043995 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:52.043955 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 09:18:52.560613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:52.559674 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 09:18:52.560613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:52.559768 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" podUID="e40f5ce9-777c-44d7-8325-6bb672235db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 09:18:53.033613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:53.033194 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 09:18:53.033613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:53.033290 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 09:18:53.561615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:53.561079 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 09:18:53.561615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:53.561147 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" podUID="e40f5ce9-777c-44d7-8325-6bb672235db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 09:18:54.040617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:54.031059 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 09:18:54.040617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:54.031129 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 09:18:54.559617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:54.559205 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 09:18:54.559617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:54.559268 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" podUID="e40f5ce9-777c-44d7-8325-6bb672235db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 09:18:54.855674 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:54.849524 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt"] Oct 30 09:18:54.860620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:54.860529 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:54.879818 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:54.879782 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt"] Oct 30 09:18:54.991221 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:54.991174 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjzmv\" (UniqueName: \"kubernetes.io/projected/f2b81448-0b6f-4093-9bdc-37139355b7e1-kube-api-access-qjzmv\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:54.991221 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:54.991225 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:54.991942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:54.991348 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b81448-0b6f-4093-9bdc-37139355b7e1-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:54.991942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:54.991447 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:54.991942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:54.991536 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:54.991942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:54.991560 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:54.991942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:54.991577 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:54.991942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:54.991667 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:54.991942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:54.991716 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.042143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.042101 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 09:18:55.066756 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.066698 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 09:18:55.092794 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.092749 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b81448-0b6f-4093-9bdc-37139355b7e1-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.093024 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.092850 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.093024 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.092934 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.093024 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.092969 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.093024 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.092992 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.093234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.093036 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.093234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.093090 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.093234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.093177 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzmv\" (UniqueName: \"kubernetes.io/projected/f2b81448-0b6f-4093-9bdc-37139355b7e1-kube-api-access-qjzmv\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.093234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.093220 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.093491 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.093461 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b81448-0b6f-4093-9bdc-37139355b7e1-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.093677 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.093658 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.093768 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.093731 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.093829 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.093792 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.094117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.094095 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.096156 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.096129 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.098128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.098095 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.134503 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.134413 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.151615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.149361 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzmv\" (UniqueName: \"kubernetes.io/projected/f2b81448-0b6f-4093-9bdc-37139355b7e1-kube-api-access-qjzmv\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.199395 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.199355 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:55.567184 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.567145 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 09:18:55.567495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.567467 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" podUID="e40f5ce9-777c-44d7-8325-6bb672235db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 09:18:55.818702 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.787524 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt"] Oct 30 09:18:55.989660 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.980074 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:55.989660 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.980182 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:55.989660 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:55.980226 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:18:56.028574 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:56.028094 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 09:18:56.028574 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:56.028174 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 09:18:56.561170 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:56.561136 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 09:18:56.561901 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:56.561794 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" podUID="e40f5ce9-777c-44d7-8325-6bb672235db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 09:18:56.644632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:56.643719 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:56.853678 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:56.853037 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:56.857655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:56.855164 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:56.947619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:56.946888 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" event={"ID":"f2b81448-0b6f-4093-9bdc-37139355b7e1","Type":"ContainerStarted","Data":"3872aa546cdf36a12365fd99d37eb06528a6b8607721d89e8925211c8e5b60c9"} Oct 30 09:18:56.947619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:56.946962 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" event={"ID":"f2b81448-0b6f-4093-9bdc-37139355b7e1","Type":"ContainerStarted","Data":"d89497ba28e195e731eb99e495ac66ff79716e841f4a8b504a6835c0f26df851"} Oct 30 09:18:57.038618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.037803 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" podStartSLOduration=3.037778361 podStartE2EDuration="3.037778361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:57.036176919 +0000 UTC m=+1454.898882743" watchObservedRunningTime="2026-10-30 09:18:57.037778361 +0000 UTC m=+1454.900484182" Oct 30 09:18:57.046614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.044358 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 09:18:57.046614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.044424 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 09:18:57.085452 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.081895 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:57.124621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.122702 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 09:18:57.149617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.148732 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:57.202766 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.200049 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:18:57.206776 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.202162 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 09:18:57.207119 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.207071 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" podUID="f2b81448-0b6f-4093-9bdc-37139355b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 09:18:57.292805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.292723 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:57.293467 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.293432 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:57.486259 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.485340 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Oct 30 09:18:57.563924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.560278 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 09:18:57.563924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.560341 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" podUID="e40f5ce9-777c-44d7-8325-6bb672235db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 09:18:57.797674 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.796618 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:57.857634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.855702 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:57.857634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.856639 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:18:57.899617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.898916 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:57.899617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:57.899609 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:18:58.006615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:58.005769 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:58.076676 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:58.059147 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" podUID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:58.235623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:58.234253 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 09:18:58.235623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:58.234320 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" podUID="f2b81448-0b6f-4093-9bdc-37139355b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 09:18:58.602615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:58.601754 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 09:18:58.602615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:58.601821 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" podUID="e40f5ce9-777c-44d7-8325-6bb672235db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 09:18:58.674624 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:58.665478 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:58.726234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:58.726176 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:18:58.726459 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:58.726252 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" podUID="00c9d38a-98f0-4598-8650-d49c9f16be07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:58.774629 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:58.774194 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:18:58.856773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:58.799844 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:58.856773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:58.800400 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:18:59.049618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:59.047882 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:59.049618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:59.048721 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:59.054212 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:59.053577 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:18:59.178965 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:59.178861 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 09:18:59.204622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:59.202965 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:59.204622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:59.203924 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:18:59.204622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:59.203980 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:59.211078 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:59.206217 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 09:18:59.211078 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:59.206281 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" podUID="f2b81448-0b6f-4093-9bdc-37139355b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 09:18:59.294074 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:59.292489 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 09:18:59.350615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:59.346650 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 09:18:59.593682 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:59.589786 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 09:18:59.613867 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:59.613791 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" podUID="e40f5ce9-777c-44d7-8325-6bb672235db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 09:18:59.786616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:59.778701 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:59.802481 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:18:59.802435 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 09:19:00.058617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:00.049412 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:19:00.058617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:00.049863 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:00.150616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:00.150556 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:19:00.210625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:00.201051 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 09:19:00.210625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:00.201114 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" podUID="f2b81448-0b6f-4093-9bdc-37139355b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 09:19:00.210625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:00.205834 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:00.210625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:00.205895 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:00.281360 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:00.281310 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:00.340298 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:00.340166 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:19:00.368942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:00.368897 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:19:00.369679 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:00.369652 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:19:00.978218 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:00.978181 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:19:00.979811 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:00.979653 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:19:00.990048 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:00.990001 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:19:01.201613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:01.200832 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 09:19:01.201613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:01.200879 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" podUID="f2b81448-0b6f-4093-9bdc-37139355b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 09:19:01.259711 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:01.256787 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:01.259711 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:01.256853 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:01.569670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:01.568022 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:19:01.569670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:01.569267 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:19:01.572616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:01.572518 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:19:02.200671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:02.200637 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 09:19:02.201085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:02.200694 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" podUID="f2b81448-0b6f-4093-9bdc-37139355b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 09:19:03.200538 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:03.200492 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 09:19:03.200785 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:03.200556 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" podUID="f2b81448-0b6f-4093-9bdc-37139355b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 09:19:04.200753 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.200677 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 09:19:04.200753 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.200745 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" podUID="f2b81448-0b6f-4093-9bdc-37139355b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 09:19:04.545135 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.545002 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg"] Oct 30 09:19:04.549299 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.549268 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg" Oct 30 09:19:04.582350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.582305 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:19:04.582350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.582307 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:19:04.585856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.585823 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg"] Oct 30 09:19:04.629628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.629558 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/515e16d8-c8ca-4e60-b1ae-462c41c614ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb41e22d-x25rg\" (UID: \"515e16d8-c8ca-4e60-b1ae-462c41c614ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg" Oct 30 09:19:04.629800 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.629633 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/515e16d8-c8ca-4e60-b1ae-462c41c614ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb41e22d-x25rg\" (UID: \"515e16d8-c8ca-4e60-b1ae-462c41c614ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg" Oct 30 09:19:04.629800 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.629760 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnq7x\" (UniqueName: \"kubernetes.io/projected/515e16d8-c8ca-4e60-b1ae-462c41c614ab-kube-api-access-rnq7x\") pod \"maas-api-revoke-sub-cb41e22d-x25rg\" (UID: \"515e16d8-c8ca-4e60-b1ae-462c41c614ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg" Oct 30 09:19:04.730951 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.730912 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/515e16d8-c8ca-4e60-b1ae-462c41c614ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb41e22d-x25rg\" (UID: \"515e16d8-c8ca-4e60-b1ae-462c41c614ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg" Oct 30 09:19:04.730951 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.730955 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/515e16d8-c8ca-4e60-b1ae-462c41c614ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb41e22d-x25rg\" (UID: \"515e16d8-c8ca-4e60-b1ae-462c41c614ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg" Oct 30 09:19:04.731263 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.730988 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnq7x\" (UniqueName: \"kubernetes.io/projected/515e16d8-c8ca-4e60-b1ae-462c41c614ab-kube-api-access-rnq7x\") pod \"maas-api-revoke-sub-cb41e22d-x25rg\" (UID: \"515e16d8-c8ca-4e60-b1ae-462c41c614ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg" Oct 30 09:19:04.731762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.731735 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/515e16d8-c8ca-4e60-b1ae-462c41c614ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb41e22d-x25rg\" (UID: \"515e16d8-c8ca-4e60-b1ae-462c41c614ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg" Oct 30 09:19:04.733570 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.733547 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/515e16d8-c8ca-4e60-b1ae-462c41c614ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb41e22d-x25rg\" (UID: \"515e16d8-c8ca-4e60-b1ae-462c41c614ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg" Oct 30 09:19:04.755375 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.755335 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnq7x\" (UniqueName: \"kubernetes.io/projected/515e16d8-c8ca-4e60-b1ae-462c41c614ab-kube-api-access-rnq7x\") pod \"maas-api-revoke-sub-cb41e22d-x25rg\" (UID: \"515e16d8-c8ca-4e60-b1ae-462c41c614ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg" Oct 30 09:19:04.864264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:04.864234 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg" Oct 30 09:19:05.198611 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:05.198563 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg"] Oct 30 09:19:05.201492 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:19:05.201463 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515e16d8_c8ca_4e60_b1ae_462c41c614ab.slice/crio-8fc96893d0113c03d4aaebf3ecb7b5cdd52fb9897bfd20f37b6bcb6ff1216bc2 WatchSource:0}: Error finding container 8fc96893d0113c03d4aaebf3ecb7b5cdd52fb9897bfd20f37b6bcb6ff1216bc2: Status 404 returned error can't find the container with id 8fc96893d0113c03d4aaebf3ecb7b5cdd52fb9897bfd20f37b6bcb6ff1216bc2 Oct 30 09:19:05.613234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:05.613112 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:19:05.614135 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:05.614104 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:19:05.615189 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:05.615031 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 09:19:05.632524 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:05.632490 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg" event={"ID":"515e16d8-c8ca-4e60-b1ae-462c41c614ab","Type":"ContainerStarted","Data":"8fc96893d0113c03d4aaebf3ecb7b5cdd52fb9897bfd20f37b6bcb6ff1216bc2"} Oct 30 09:19:06.639935 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:06.639895 5938 generic.go:356] "Generic (PLEG): container finished" podID="515e16d8-c8ca-4e60-b1ae-462c41c614ab" containerID="8eafd18cee6cae022e8b3d2bb75979fee4429aabf486a6971296a320fc4fd959" exitCode=0 Oct 30 09:19:06.640320 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:06.639938 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg" event={"ID":"515e16d8-c8ca-4e60-b1ae-462c41c614ab","Type":"ContainerDied","Data":"8eafd18cee6cae022e8b3d2bb75979fee4429aabf486a6971296a320fc4fd959"} Oct 30 09:19:07.809870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:07.809840 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg" Oct 30 09:19:07.885394 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:07.885352 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/515e16d8-c8ca-4e60-b1ae-462c41c614ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/515e16d8-c8ca-4e60-b1ae-462c41c614ab-maas-api-xks-ca\") pod \"515e16d8-c8ca-4e60-b1ae-462c41c614ab\" (UID: \"515e16d8-c8ca-4e60-b1ae-462c41c614ab\") " Oct 30 09:19:07.885394 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:07.885397 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/515e16d8-c8ca-4e60-b1ae-462c41c614ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/515e16d8-c8ca-4e60-b1ae-462c41c614ab-maas-api-openshift-ca\") pod \"515e16d8-c8ca-4e60-b1ae-462c41c614ab\" (UID: \"515e16d8-c8ca-4e60-b1ae-462c41c614ab\") " Oct 30 09:19:07.885687 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:07.885600 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/515e16d8-c8ca-4e60-b1ae-462c41c614ab-kube-api-access-rnq7x\" (UniqueName: \"kubernetes.io/projected/515e16d8-c8ca-4e60-b1ae-462c41c614ab-kube-api-access-rnq7x\") pod \"515e16d8-c8ca-4e60-b1ae-462c41c614ab\" (UID: \"515e16d8-c8ca-4e60-b1ae-462c41c614ab\") " Oct 30 09:19:07.885911 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:07.885882 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515e16d8-c8ca-4e60-b1ae-462c41c614ab-maas-api-openshift-ca" pod "515e16d8-c8ca-4e60-b1ae-462c41c614ab" (UID: "515e16d8-c8ca-4e60-b1ae-462c41c614ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:19:07.886176 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:07.886155 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/515e16d8-c8ca-4e60-b1ae-462c41c614ab-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:07.888011 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:07.887978 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/515e16d8-c8ca-4e60-b1ae-462c41c614ab-maas-api-xks-ca" pod "515e16d8-c8ca-4e60-b1ae-462c41c614ab" (UID: "515e16d8-c8ca-4e60-b1ae-462c41c614ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:19:07.888011 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:07.887995 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515e16d8-c8ca-4e60-b1ae-462c41c614ab-kube-api-access-rnq7x" pod "515e16d8-c8ca-4e60-b1ae-462c41c614ab" (UID: "515e16d8-c8ca-4e60-b1ae-462c41c614ab"). InnerVolumeSpecName "kube-api-access-rnq7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:19:07.987898 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:07.987858 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnq7x\" (UniqueName: \"kubernetes.io/projected/515e16d8-c8ca-4e60-b1ae-462c41c614ab-kube-api-access-rnq7x\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:07.987898 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:07.987893 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/515e16d8-c8ca-4e60-b1ae-462c41c614ab-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:08.653693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:08.653666 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg" Oct 30 09:19:08.653883 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:08.653703 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg" event={"ID":"515e16d8-c8ca-4e60-b1ae-462c41c614ab","Type":"ContainerDied","Data":"8fc96893d0113c03d4aaebf3ecb7b5cdd52fb9897bfd20f37b6bcb6ff1216bc2"} Oct 30 09:19:08.653883 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:08.653752 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fc96893d0113c03d4aaebf3ecb7b5cdd52fb9897bfd20f37b6bcb6ff1216bc2" Oct 30 09:19:15.814362 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:19:15.814315 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:19:20.896621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:20.890063 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc"] Oct 30 09:19:20.918618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:20.902500 5938 cpu_manager.go:414] "containerMap: removing container" podUID="515e16d8-c8ca-4e60-b1ae-462c41c614ab" containerName="revoke-keys" Oct 30 09:19:20.918618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:20.902544 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="515e16d8-c8ca-4e60-b1ae-462c41c614ab" containerName="revoke-keys" Oct 30 09:19:20.918618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:20.902975 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="515e16d8-c8ca-4e60-b1ae-462c41c614ab" containerName="revoke-keys" Oct 30 09:19:20.963615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:20.952952 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc"] Oct 30 09:19:20.963615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:20.953135 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.025209 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.025161 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:21.027291 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.025875 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" podUID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 09:19:21.088651 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.088573 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:21.088840 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.088722 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 09:19:21.133392 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.133262 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz"] Oct 30 09:19:21.137903 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.137765 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" podUID="e1c99ffc-1e18-49c3-8327-dbcc3e939a95" containerName="istio-proxy" containerID="cri-o://108489d5d48e437df8af8ef7bb71e72c28368553390422c9f0846a9e7bae99c1" gracePeriod=30 Oct 30 09:19:21.168613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.166741 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 09:19:21.182719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.170189 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.182719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.170309 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.182719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.170343 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.182719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.170432 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.182719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.170488 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9549w\" (UniqueName: \"kubernetes.io/projected/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-kube-api-access-9549w\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.182719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.170528 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.182719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.170631 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.182719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.170709 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.182719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.170757 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.288625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.288565 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.296634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.289546 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9549w\" (UniqueName: \"kubernetes.io/projected/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-kube-api-access-9549w\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.296634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.289622 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.296634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.289708 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.296634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.289791 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.296634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.289836 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.296634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.289909 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.296634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.290001 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.296634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.290029 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.296634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.290575 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.311096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.309875 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:21.361635 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.333120 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9549w\" (UniqueName: \"kubernetes.io/projected/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-kube-api-access-9549w\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.361635 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.343177 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.361635 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.343779 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6"] Oct 30 09:19:21.361635 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.344103 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.361635 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.344360 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.361635 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.359046 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.386778 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.368386 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" podUID="26d8a515-9a87-4cf5-9ec3-08fda2172bcc" containerName="istio-proxy" containerID="cri-o://8f1efd018c309b64127d8d4489a42098fb79ccf858b5f56aecdd6b4265d9c70a" gracePeriod=30 Oct 30 09:19:21.423750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.288999 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 09:19:21.423750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.390155 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:21.423750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.391683 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.800552 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.796477 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.800552 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.798955 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:21.840618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.835200 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:21.864751 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.864699 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:21.992269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:21.992231 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:22.043499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.043459 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:22.044461 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.044344 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:19:22.099632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.092907 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:22.099632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.094646 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:19:22.101448 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.100559 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 09:19:22.102700 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.101770 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:19:22.176851 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.174886 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:22.318680 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.313211 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:22.396619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.392712 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:22.396619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.393148 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 09:19:22.415169 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.415140 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:19:22.415429 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.415412 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:19:22.415570 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.415558 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:19:22.450149 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.450087 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:19:22.658544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.658500 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:22.678089 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.677991 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc"] Oct 30 09:19:22.720359 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:19:22.720324 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2b8ea1_4db6_479d_b681_a9a26b02ddec.slice/crio-4819277cbd40df0d6383de2a5632beb3a8faf18cb3e5e37f40166c64b4791034 WatchSource:0}: Error finding container 4819277cbd40df0d6383de2a5632beb3a8faf18cb3e5e37f40166c64b4791034: Status 404 returned error can't find the container with id 4819277cbd40df0d6383de2a5632beb3a8faf18cb3e5e37f40166c64b4791034 Oct 30 09:19:22.827616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.812474 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:22.827616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.813424 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:19:22.904689 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.904079 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:19:22.904689 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.904192 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:19:22.904689 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.904243 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:19:22.924645 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:22.924612 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" event={"ID":"7a2b8ea1-4db6-479d-b681-a9a26b02ddec","Type":"ContainerStarted","Data":"4819277cbd40df0d6383de2a5632beb3a8faf18cb3e5e37f40166c64b4791034"} Oct 30 09:19:23.054221 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:23.053167 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 09:19:23.106044 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:23.106001 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:23.273918 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:23.273830 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:19:23.276644 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:23.276618 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:19:23.305758 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:23.305729 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:19:23.317918 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:23.317875 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:23.438658 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:23.426712 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:23.438658 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:23.429359 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 09:19:23.793850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:23.793809 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:19:24.175617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:24.170437 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:24.318951 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:24.318904 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 09:19:24.464218 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:24.443956 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:24.464218 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:24.455558 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:19:24.517004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:24.510145 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:19:25.039962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:25.039877 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" event={"ID":"7a2b8ea1-4db6-479d-b681-a9a26b02ddec","Type":"ContainerStarted","Data":"558ab888bd950576602ca02dab19593da8d34ca1cd3e111e0fecc01a606fc631"} Oct 30 09:19:25.126169 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:25.126110 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" podStartSLOduration=5.126089821 podStartE2EDuration="5.126089821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:25.110864972 +0000 UTC m=+1482.973570792" watchObservedRunningTime="2026-10-30 09:19:25.126089821 +0000 UTC m=+1482.988795645" Oct 30 09:19:25.994335 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:25.994296 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:25.996135 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:25.996108 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 09:19:25.996256 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:25.996162 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" podUID="7a2b8ea1-4db6-479d-b681-a9a26b02ddec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 09:19:26.552027 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.551955 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:19:26.647152 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.647116 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-workload-certs\") pod \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " Oct 30 09:19:26.647346 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.647198 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-podinfo\") pod \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " Oct 30 09:19:26.647346 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.647315 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-envoy\") pod \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " Oct 30 09:19:26.647496 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.647352 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-workload-socket\") pod \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " Oct 30 09:19:26.647496 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.647401 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istiod-ca-cert\") pod \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " Oct 30 09:19:26.647496 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.647442 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-credential-socket\") pod \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " Oct 30 09:19:26.647691 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.647521 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-data\") pod \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " Oct 30 09:19:26.647691 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.647566 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-token\" (UniqueName: \"kubernetes.io/projected/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-token\") pod \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " Oct 30 09:19:26.647691 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.647660 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-kube-api-access-vhkg8\" (UniqueName: \"kubernetes.io/projected/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-kube-api-access-vhkg8\") pod \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\" (UID: \"e1c99ffc-1e18-49c3-8327-dbcc3e939a95\") " Oct 30 09:19:26.648031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.647958 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-workload-socket" pod "e1c99ffc-1e18-49c3-8327-dbcc3e939a95" (UID: "e1c99ffc-1e18-49c3-8327-dbcc3e939a95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:19:26.648031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.647987 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-credential-socket" pod "e1c99ffc-1e18-49c3-8327-dbcc3e939a95" (UID: "e1c99ffc-1e18-49c3-8327-dbcc3e939a95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:19:26.648373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.648214 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-data" pod "e1c99ffc-1e18-49c3-8327-dbcc3e939a95" (UID: "e1c99ffc-1e18-49c3-8327-dbcc3e939a95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:19:26.648373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.648302 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istiod-ca-cert" pod "e1c99ffc-1e18-49c3-8327-dbcc3e939a95" (UID: "e1c99ffc-1e18-49c3-8327-dbcc3e939a95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:19:26.648516 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.648409 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-workload-certs" pod "e1c99ffc-1e18-49c3-8327-dbcc3e939a95" (UID: "e1c99ffc-1e18-49c3-8327-dbcc3e939a95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:19:26.648904 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.648663 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:26.648904 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.648700 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:26.648904 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.648715 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:26.654015 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.653985 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-envoy" pod "e1c99ffc-1e18-49c3-8327-dbcc3e939a95" (UID: "e1c99ffc-1e18-49c3-8327-dbcc3e939a95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:19:26.654536 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.654460 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-kube-api-access-vhkg8" pod "e1c99ffc-1e18-49c3-8327-dbcc3e939a95" (UID: "e1c99ffc-1e18-49c3-8327-dbcc3e939a95"). InnerVolumeSpecName "kube-api-access-vhkg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:19:26.655065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.655016 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-podinfo" pod "e1c99ffc-1e18-49c3-8327-dbcc3e939a95" (UID: "e1c99ffc-1e18-49c3-8327-dbcc3e939a95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:19:26.656247 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.656221 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-token" pod "e1c99ffc-1e18-49c3-8327-dbcc3e939a95" (UID: "e1c99ffc-1e18-49c3-8327-dbcc3e939a95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:19:26.750623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.750546 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:26.750623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.750626 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:26.750862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.750641 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:26.750862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.750654 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:26.750862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.750667 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:26.750862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.750679 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhkg8\" (UniqueName: \"kubernetes.io/projected/e1c99ffc-1e18-49c3-8327-dbcc3e939a95-kube-api-access-vhkg8\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:26.994922 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.994000 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 09:19:26.994922 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:26.994045 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" podUID="7a2b8ea1-4db6-479d-b681-a9a26b02ddec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 09:19:27.074867 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.074828 5938 generic.go:356] "Generic (PLEG): container finished" podID="26d8a515-9a87-4cf5-9ec3-08fda2172bcc" containerID="8f1efd018c309b64127d8d4489a42098fb79ccf858b5f56aecdd6b4265d9c70a" exitCode=0 Oct 30 09:19:27.075092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.074908 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" event={"ID":"26d8a515-9a87-4cf5-9ec3-08fda2172bcc","Type":"ContainerDied","Data":"8f1efd018c309b64127d8d4489a42098fb79ccf858b5f56aecdd6b4265d9c70a"} Oct 30 09:19:27.076690 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.076658 5938 generic.go:356] "Generic (PLEG): container finished" podID="e1c99ffc-1e18-49c3-8327-dbcc3e939a95" containerID="108489d5d48e437df8af8ef7bb71e72c28368553390422c9f0846a9e7bae99c1" exitCode=0 Oct 30 09:19:27.083916 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.077693 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" event={"ID":"e1c99ffc-1e18-49c3-8327-dbcc3e939a95","Type":"ContainerDied","Data":"108489d5d48e437df8af8ef7bb71e72c28368553390422c9f0846a9e7bae99c1"} Oct 30 09:19:27.083916 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.077774 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" event={"ID":"e1c99ffc-1e18-49c3-8327-dbcc3e939a95","Type":"ContainerDied","Data":"71f5b9d726313ce0af28f9363a5285db0c93f93692e7c212efceb2a836ff8c81"} Oct 30 09:19:27.083916 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.077717 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz" Oct 30 09:19:27.083916 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.077831 5938 scope.go:122] "RemoveContainer" containerID="108489d5d48e437df8af8ef7bb71e72c28368553390422c9f0846a9e7bae99c1" Oct 30 09:19:27.092328 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.092305 5938 scope.go:122] "RemoveContainer" containerID="108489d5d48e437df8af8ef7bb71e72c28368553390422c9f0846a9e7bae99c1" Oct 30 09:19:27.092742 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:19:27.092717 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"108489d5d48e437df8af8ef7bb71e72c28368553390422c9f0846a9e7bae99c1\": container with ID starting with 108489d5d48e437df8af8ef7bb71e72c28368553390422c9f0846a9e7bae99c1 not found: ID does not exist" containerID="108489d5d48e437df8af8ef7bb71e72c28368553390422c9f0846a9e7bae99c1" Oct 30 09:19:27.092858 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.092746 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"108489d5d48e437df8af8ef7bb71e72c28368553390422c9f0846a9e7bae99c1"} err="failed to get container status \"108489d5d48e437df8af8ef7bb71e72c28368553390422c9f0846a9e7bae99c1\": rpc error: code = NotFound desc = could not find container \"108489d5d48e437df8af8ef7bb71e72c28368553390422c9f0846a9e7bae99c1\": container with ID starting with 108489d5d48e437df8af8ef7bb71e72c28368553390422c9f0846a9e7bae99c1 not found: ID does not exist" Oct 30 09:19:27.179651 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.178731 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz"] Oct 30 09:19:27.194496 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.194457 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2hjgz"] Oct 30 09:19:27.393119 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.393084 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:19:27.396960 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.396932 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-workload-socket\") pod \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " Oct 30 09:19:27.397185 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.396973 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-data\") pod \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " Oct 30 09:19:27.397185 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.397039 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istiod-ca-cert\") pod \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " Oct 30 09:19:27.397185 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.397108 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-envoy\") pod \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " Oct 30 09:19:27.397185 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.397149 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-kube-api-access-lw59s\" (UniqueName: \"kubernetes.io/projected/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-kube-api-access-lw59s\") pod \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " Oct 30 09:19:27.397413 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.397195 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-podinfo\") pod \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " Oct 30 09:19:27.397413 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.397236 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-token\" (UniqueName: \"kubernetes.io/projected/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-token\") pod \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " Oct 30 09:19:27.397413 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.397297 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-credential-socket\") pod \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " Oct 30 09:19:27.397413 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.397345 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-workload-certs\") pod \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\" (UID: \"26d8a515-9a87-4cf5-9ec3-08fda2172bcc\") " Oct 30 09:19:27.401238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.398626 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-data" pod "26d8a515-9a87-4cf5-9ec3-08fda2172bcc" (UID: "26d8a515-9a87-4cf5-9ec3-08fda2172bcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:19:27.401238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.398798 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-workload-socket" pod "26d8a515-9a87-4cf5-9ec3-08fda2172bcc" (UID: "26d8a515-9a87-4cf5-9ec3-08fda2172bcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:19:27.401238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.398853 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-workload-certs" pod "26d8a515-9a87-4cf5-9ec3-08fda2172bcc" (UID: "26d8a515-9a87-4cf5-9ec3-08fda2172bcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:19:27.401238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.399121 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-credential-socket" pod "26d8a515-9a87-4cf5-9ec3-08fda2172bcc" (UID: "26d8a515-9a87-4cf5-9ec3-08fda2172bcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:19:27.401238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.401148 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istiod-ca-cert" pod "26d8a515-9a87-4cf5-9ec3-08fda2172bcc" (UID: "26d8a515-9a87-4cf5-9ec3-08fda2172bcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:19:27.401790 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.401313 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-token" pod "26d8a515-9a87-4cf5-9ec3-08fda2172bcc" (UID: "26d8a515-9a87-4cf5-9ec3-08fda2172bcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:19:27.401790 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.401389 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-podinfo" pod "26d8a515-9a87-4cf5-9ec3-08fda2172bcc" (UID: "26d8a515-9a87-4cf5-9ec3-08fda2172bcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:19:27.401923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.401899 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-kube-api-access-lw59s" pod "26d8a515-9a87-4cf5-9ec3-08fda2172bcc" (UID: "26d8a515-9a87-4cf5-9ec3-08fda2172bcc"). InnerVolumeSpecName "kube-api-access-lw59s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:19:27.403892 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.403863 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-envoy" pod "26d8a515-9a87-4cf5-9ec3-08fda2172bcc" (UID: "26d8a515-9a87-4cf5-9ec3-08fda2172bcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:19:27.499670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.499626 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:27.499670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.499664 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:27.499670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.499682 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:27.500010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.499696 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:27.500010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.499711 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw59s\" (UniqueName: \"kubernetes.io/projected/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-kube-api-access-lw59s\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:27.500010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.499723 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:27.500010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.499736 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:27.500010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.499749 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:27.500010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.499762 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d8a515-9a87-4cf5-9ec3-08fda2172bcc-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:19:27.994237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.994194 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 09:19:27.994438 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:27.994263 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" podUID="7a2b8ea1-4db6-479d-b681-a9a26b02ddec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 09:19:28.086165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:28.086134 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" Oct 30 09:19:28.086699 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:28.086162 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6" event={"ID":"26d8a515-9a87-4cf5-9ec3-08fda2172bcc","Type":"ContainerDied","Data":"66a8bdbb4fc749459558e97e3d820ccc122638741dd0239904beb21660a3d012"} Oct 30 09:19:28.086699 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:28.086235 5938 scope.go:122] "RemoveContainer" containerID="8f1efd018c309b64127d8d4489a42098fb79ccf858b5f56aecdd6b4265d9c70a" Oct 30 09:19:28.152353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:28.152306 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6"] Oct 30 09:19:28.155241 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:28.155211 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v8xs6"] Oct 30 09:19:28.267706 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:28.267572 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d8a515-9a87-4cf5-9ec3-08fda2172bcc" path="/var/lib/kubelet/pods/26d8a515-9a87-4cf5-9ec3-08fda2172bcc/volumes" Oct 30 09:19:28.268134 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:28.268118 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1c99ffc-1e18-49c3-8327-dbcc3e939a95" path="/var/lib/kubelet/pods/e1c99ffc-1e18-49c3-8327-dbcc3e939a95/volumes" Oct 30 09:19:28.994193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:28.994134 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 09:19:28.994505 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:28.994267 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" podUID="7a2b8ea1-4db6-479d-b681-a9a26b02ddec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 09:19:29.994565 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:29.994529 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 09:19:29.995057 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:29.994630 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" podUID="7a2b8ea1-4db6-479d-b681-a9a26b02ddec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 09:19:30.993509 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:30.993471 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 09:19:30.993717 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:30.993541 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" podUID="7a2b8ea1-4db6-479d-b681-a9a26b02ddec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 09:19:31.283819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.283716 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c"] Oct 30 09:19:31.284876 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.284850 5938 cpu_manager.go:414] "containerMap: removing container" podUID="26d8a515-9a87-4cf5-9ec3-08fda2172bcc" containerName="istio-proxy" Oct 30 09:19:31.284876 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.284876 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d8a515-9a87-4cf5-9ec3-08fda2172bcc" containerName="istio-proxy" Oct 30 09:19:31.285087 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.284975 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e1c99ffc-1e18-49c3-8327-dbcc3e939a95" containerName="istio-proxy" Oct 30 09:19:31.285087 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.284987 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1c99ffc-1e18-49c3-8327-dbcc3e939a95" containerName="istio-proxy" Oct 30 09:19:31.285223 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.285209 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1c99ffc-1e18-49c3-8327-dbcc3e939a95" containerName="istio-proxy" Oct 30 09:19:31.285287 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.285229 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d8a515-9a87-4cf5-9ec3-08fda2172bcc" containerName="istio-proxy" Oct 30 09:19:31.289890 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.289861 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:19:31.313767 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.313730 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c"] Oct 30 09:19:31.316721 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.316691 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8eada0cf-serving-cert\"" Oct 30 09:19:31.375780 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.375737 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc0c0552-4af6-429f-bda5-bf438f8703a6-metrics-certs\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c\" (UID: \"dc0c0552-4af6-429f-bda5-bf438f8703a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:19:31.375954 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.375911 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs5wv\" (UniqueName: \"kubernetes.io/projected/dc0c0552-4af6-429f-bda5-bf438f8703a6-kube-api-access-cs5wv\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c\" (UID: \"dc0c0552-4af6-429f-bda5-bf438f8703a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:19:31.376028 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.376016 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc0c0552-4af6-429f-bda5-bf438f8703a6-openshift-service-ca\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c\" (UID: \"dc0c0552-4af6-429f-bda5-bf438f8703a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:19:31.376235 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.376208 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc0c0552-4af6-429f-bda5-bf438f8703a6-maas-api-tls\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c\" (UID: \"dc0c0552-4af6-429f-bda5-bf438f8703a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:19:31.477719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.477684 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc0c0552-4af6-429f-bda5-bf438f8703a6-maas-api-tls\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c\" (UID: \"dc0c0552-4af6-429f-bda5-bf438f8703a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:19:31.477925 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.477751 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc0c0552-4af6-429f-bda5-bf438f8703a6-metrics-certs\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c\" (UID: \"dc0c0552-4af6-429f-bda5-bf438f8703a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:19:31.477925 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.477900 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs5wv\" (UniqueName: \"kubernetes.io/projected/dc0c0552-4af6-429f-bda5-bf438f8703a6-kube-api-access-cs5wv\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c\" (UID: \"dc0c0552-4af6-429f-bda5-bf438f8703a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:19:31.479452 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.477988 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc0c0552-4af6-429f-bda5-bf438f8703a6-openshift-service-ca\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c\" (UID: \"dc0c0552-4af6-429f-bda5-bf438f8703a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:19:31.479452 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.479153 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc0c0552-4af6-429f-bda5-bf438f8703a6-openshift-service-ca\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c\" (UID: \"dc0c0552-4af6-429f-bda5-bf438f8703a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:19:31.481495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.481431 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc0c0552-4af6-429f-bda5-bf438f8703a6-maas-api-tls\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c\" (UID: \"dc0c0552-4af6-429f-bda5-bf438f8703a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:19:31.482487 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.482457 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc0c0552-4af6-429f-bda5-bf438f8703a6-metrics-certs\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c\" (UID: \"dc0c0552-4af6-429f-bda5-bf438f8703a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:19:31.513195 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.513160 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs5wv\" (UniqueName: \"kubernetes.io/projected/dc0c0552-4af6-429f-bda5-bf438f8703a6-kube-api-access-cs5wv\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c\" (UID: \"dc0c0552-4af6-429f-bda5-bf438f8703a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:19:31.605352 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.605246 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:19:31.926818 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:31.926790 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c"] Oct 30 09:19:31.929743 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:19:31.929712 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0c0552_4af6_429f_bda5_bf438f8703a6.slice/crio-e5d1c1cac14645806ff4c79cc7774c04ed6d7151fc5116dccfa194304170ee21 WatchSource:0}: Error finding container e5d1c1cac14645806ff4c79cc7774c04ed6d7151fc5116dccfa194304170ee21: Status 404 returned error can't find the container with id e5d1c1cac14645806ff4c79cc7774c04ed6d7151fc5116dccfa194304170ee21 Oct 30 09:19:32.126457 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:32.126415 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" event={"ID":"dc0c0552-4af6-429f-bda5-bf438f8703a6","Type":"ContainerStarted","Data":"e5d1c1cac14645806ff4c79cc7774c04ed6d7151fc5116dccfa194304170ee21"} Oct 30 09:19:32.456009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:32.455972 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:32.457671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:32.457645 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:32.458816 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:32.458780 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:19:33.205908 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:33.205858 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" event={"ID":"dc0c0552-4af6-429f-bda5-bf438f8703a6","Type":"ContainerStarted","Data":"860a9bd4c21ad5c66d1cb4617a25c229506e64e40a77febf031c6aad2745e5c0"} Oct 30 09:19:33.253889 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:33.253834 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" podStartSLOduration=1.8338529430000001 podStartE2EDuration="2.253818832s" totalImagesPullingTime="419.965889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:33.251617393 +0000 UTC m=+1491.114323214" watchObservedRunningTime="2026-10-30 09:19:33.253818832 +0000 UTC m=+1491.116524652" Oct 30 09:19:34.213071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.213034 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:19:34.380515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.380405 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9"] Oct 30 09:19:34.386749 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.386718 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.421074 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.421015 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.421074 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.421073 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.421318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.421193 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.421318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.421265 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.421318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.421306 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.421490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.421405 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.421547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.421532 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.421703 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.421683 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.421793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.421722 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hh8j\" (UniqueName: \"kubernetes.io/projected/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-kube-api-access-7hh8j\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.424659 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.424625 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9"] Oct 30 09:19:34.522845 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.522726 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.522845 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.522796 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.522845 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.522833 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.523168 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.522928 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.523168 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.523019 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.523168 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.523052 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hh8j\" (UniqueName: \"kubernetes.io/projected/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-kube-api-access-7hh8j\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.523168 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.523134 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.523381 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.523169 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.523381 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.523205 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.523662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.523637 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.523757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.523705 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.524130 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.524103 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.524410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.524388 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.524515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.524491 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.525987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.525962 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.527021 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.527000 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.545133 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.545094 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.545317 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.545232 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hh8j\" (UniqueName: \"kubernetes.io/projected/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-kube-api-access-7hh8j\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:34.728525 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:34.728478 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:35.061683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:35.061653 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9"] Oct 30 09:19:35.063045 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:19:35.063014 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0ba647_3cd7_49c1_9c5e_6872fd5a0fa1.slice/crio-8d9b4306e479b7231be906f9801aaa50e38faf88cd867ab5f09aca83e4f2df46 WatchSource:0}: Error finding container 8d9b4306e479b7231be906f9801aaa50e38faf88cd867ab5f09aca83e4f2df46: Status 404 returned error can't find the container with id 8d9b4306e479b7231be906f9801aaa50e38faf88cd867ab5f09aca83e4f2df46 Oct 30 09:19:35.065932 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:35.065898 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:19:35.066017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:35.065968 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:19:35.066017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:35.065998 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:19:35.223004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:35.222958 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" event={"ID":"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1","Type":"ContainerStarted","Data":"a93a339ea2ca46429c22c24bbe35df2970b5eda6c52d6efcf8e1988ddb1aa504"} Oct 30 09:19:35.223509 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:35.223040 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" event={"ID":"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1","Type":"ContainerStarted","Data":"8d9b4306e479b7231be906f9801aaa50e38faf88cd867ab5f09aca83e4f2df46"} Oct 30 09:19:35.270290 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:35.270230 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" podStartSLOduration=1.270213319 podStartE2EDuration="1.270213319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:35.268985948 +0000 UTC m=+1493.131691779" watchObservedRunningTime="2026-10-30 09:19:35.270213319 +0000 UTC m=+1493.132919141" Oct 30 09:19:35.730613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:35.728683 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:36.743813 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:36.743768 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 09:19:36.744274 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:36.743895 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" podUID="7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 09:19:37.739614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:37.730716 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 09:19:37.739614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:37.730780 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" podUID="7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 09:19:38.729630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:38.729091 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 09:19:38.729630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:38.729158 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" podUID="7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 09:19:39.744089 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:39.744053 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 09:19:39.744722 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:39.744691 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" podUID="7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 09:19:40.248613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.247820 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:19:40.259830 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.258978 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547"] Oct 30 09:19:40.268924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.268882 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc"] Oct 30 09:19:40.269598 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.269007 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.274199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.274171 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547"] Oct 30 09:19:40.274527 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.274498 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.289389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.289357 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc"] Oct 30 09:19:40.308510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.308447 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.308510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.308494 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.308771 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.308536 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.308771 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.308573 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929a1bc8-672b-49b2-8344-9253178bd699-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.308771 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.308647 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.308771 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.308672 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.308771 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.308727 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zrc8\" (UniqueName: \"kubernetes.io/projected/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-kube-api-access-6zrc8\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.308771 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.308756 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.309086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.308831 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.309086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.308883 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.309086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.308915 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929a1bc8-672b-49b2-8344-9253178bd699-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.309086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.308953 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929a1bc8-672b-49b2-8344-9253178bd699-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.309086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.309026 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.309086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.309057 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.309386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.309099 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.309386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.309129 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.309386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.309164 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnzcr\" (UniqueName: \"kubernetes.io/projected/929a1bc8-672b-49b2-8344-9253178bd699-kube-api-access-dnzcr\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.309386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.309257 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.410856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.410817 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.410856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.410869 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.411135 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.410906 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.411135 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.410929 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.411135 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.410958 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnzcr\" (UniqueName: \"kubernetes.io/projected/929a1bc8-672b-49b2-8344-9253178bd699-kube-api-access-dnzcr\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.411135 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.411014 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.411135 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.411074 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.411135 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.411100 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.411442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.411140 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.411442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.411177 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929a1bc8-672b-49b2-8344-9253178bd699-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.411442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.411221 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.411442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.411250 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.411442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.411308 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zrc8\" (UniqueName: \"kubernetes.io/projected/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-kube-api-access-6zrc8\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.411442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.411342 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.411442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.411406 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.411944 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.411464 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.411944 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.411498 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929a1bc8-672b-49b2-8344-9253178bd699-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.411944 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.411540 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929a1bc8-672b-49b2-8344-9253178bd699-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.412351 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.412327 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.412458 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.412413 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929a1bc8-672b-49b2-8344-9253178bd699-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.413086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.413058 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.413178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.413123 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.414464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.413614 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.414464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.414147 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.414464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.414398 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.414772 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.414748 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.415570 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.414796 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.415570 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.414960 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.415570 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.415084 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.415570 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.415507 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.416933 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.416909 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929a1bc8-672b-49b2-8344-9253178bd699-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.417270 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.417244 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.455609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.455555 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnzcr\" (UniqueName: \"kubernetes.io/projected/929a1bc8-672b-49b2-8344-9253178bd699-kube-api-access-dnzcr\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.456250 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.456226 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929a1bc8-672b-49b2-8344-9253178bd699-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.457104 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.457060 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zrc8\" (UniqueName: \"kubernetes.io/projected/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-kube-api-access-6zrc8\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.458497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.458474 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.605872 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.605778 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:40.630252 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.628886 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:40.752304 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.752095 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 09:19:40.752304 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.752147 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" podUID="7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 09:19:40.986988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.986468 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc"] Oct 30 09:19:40.990695 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.989569 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547"] Oct 30 09:19:40.990882 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:19:40.990805 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929a1bc8_672b_49b2_8344_9253178bd699.slice/crio-1ecc0290fbc970579859cabc537012dbec44ab764ed87421772c6833ee515989 WatchSource:0}: Error finding container 1ecc0290fbc970579859cabc537012dbec44ab764ed87421772c6833ee515989: Status 404 returned error can't find the container with id 1ecc0290fbc970579859cabc537012dbec44ab764ed87421772c6833ee515989 Oct 30 09:19:40.994619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.994558 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:19:40.994779 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.994758 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:19:40.994835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:40.994824 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:19:41.003713 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:19:41.003680 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0317fa91_3439_42b9_9b92_7ff6b5f8c31a.slice/crio-618baf2da57d35de349da2b65149410dd287c83b216913108ecd1cbce0db6136 WatchSource:0}: Error finding container 618baf2da57d35de349da2b65149410dd287c83b216913108ecd1cbce0db6136: Status 404 returned error can't find the container with id 618baf2da57d35de349da2b65149410dd287c83b216913108ecd1cbce0db6136 Oct 30 09:19:41.009395 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:41.008955 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:19:41.009395 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:41.009026 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:19:41.009395 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:41.009054 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:19:41.301377 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:41.300900 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" event={"ID":"0317fa91-3439-42b9-9b92-7ff6b5f8c31a","Type":"ContainerStarted","Data":"f0569446e9f952d35b24f776b142855f8169a78697abd78b1f64a2e4737c8ed3"} Oct 30 09:19:41.301377 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:41.300960 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" event={"ID":"0317fa91-3439-42b9-9b92-7ff6b5f8c31a","Type":"ContainerStarted","Data":"618baf2da57d35de349da2b65149410dd287c83b216913108ecd1cbce0db6136"} Oct 30 09:19:41.303162 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:41.303128 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" event={"ID":"929a1bc8-672b-49b2-8344-9253178bd699","Type":"ContainerStarted","Data":"0e9c7d5f552a1be49b2ea07326a99317b554c810cc2654b3e766994d73e4267d"} Oct 30 09:19:41.303284 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:41.303191 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" event={"ID":"929a1bc8-672b-49b2-8344-9253178bd699","Type":"ContainerStarted","Data":"1ecc0290fbc970579859cabc537012dbec44ab764ed87421772c6833ee515989"} Oct 30 09:19:41.348818 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:41.348743 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" podStartSLOduration=1.348716859 podStartE2EDuration="1.348716859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:41.345926413 +0000 UTC m=+1499.208632236" watchObservedRunningTime="2026-10-30 09:19:41.348716859 +0000 UTC m=+1499.211422681" Oct 30 09:19:41.392230 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:41.392074 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" podStartSLOduration=1.392042601 podStartE2EDuration="1.392042601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:41.389395063 +0000 UTC m=+1499.252100886" watchObservedRunningTime="2026-10-30 09:19:41.392042601 +0000 UTC m=+1499.254748423" Oct 30 09:19:41.606127 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:41.606029 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:41.629306 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:41.629268 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:41.729606 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:41.729550 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 09:19:41.729815 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:41.729642 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" podUID="7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 09:19:42.608301 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:42.608269 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 09:19:42.608778 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:42.608749 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 09:19:42.643614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:42.643431 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 09:19:42.643614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:42.643494 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" podUID="929a1bc8-672b-49b2-8344-9253178bd699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 09:19:43.074616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:43.073179 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:43.074616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:43.074448 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:43.076650 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:43.075991 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:19:43.606505 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:43.606468 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 09:19:43.606763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:43.606544 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 09:19:43.654615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:43.640020 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 09:19:43.654615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:43.640078 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" podUID="929a1bc8-672b-49b2-8344-9253178bd699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 09:19:44.606754 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:44.606719 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 09:19:44.606912 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:44.606777 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 09:19:44.639668 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:44.630564 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 09:19:44.639668 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:44.630646 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" podUID="929a1bc8-672b-49b2-8344-9253178bd699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 09:19:45.606991 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:45.606952 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 09:19:45.607501 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:45.607013 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 09:19:45.643614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:45.640038 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 09:19:45.643614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:45.640106 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" podUID="929a1bc8-672b-49b2-8344-9253178bd699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 09:19:46.001541 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:19:46.001498 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:19:46.609300 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:46.609252 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 09:19:46.610136 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:46.610093 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 09:19:46.629819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:46.629772 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 09:19:46.630161 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:46.630128 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" podUID="929a1bc8-672b-49b2-8344-9253178bd699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 09:19:47.319778 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:47.319699 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:47.332748 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:47.332700 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:47.346615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:47.338707 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:47.347969 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:47.347928 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:47.459540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:47.457709 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:47.513809 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:47.513696 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:47.620207 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:47.620168 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 09:19:47.620874 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:47.620823 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 09:19:47.638211 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:47.638163 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 09:19:47.638611 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:47.638540 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" podUID="929a1bc8-672b-49b2-8344-9253178bd699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 09:19:47.795763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:47.795711 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 09:19:48.004213 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.004105 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" podUID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 09:19:48.109758 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.109702 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:48.159758 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.159706 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:48.340819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.340705 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 09:19:48.341914 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.341886 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:19:48.348744 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.348694 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:48.349459 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.349435 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:19:48.506764 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.506715 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 09:19:48.761632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.537729 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 09:19:48.762752 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.663382 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 09:19:48.762993 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.762948 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 09:19:48.763161 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.663814 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": context deadline exceeded" Oct 30 09:19:48.763286 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.663771 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 09:19:48.784957 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.784925 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:19:48.811442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.811393 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:48.812205 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.812151 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:19:48.824029 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.823981 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" podUID="929a1bc8-672b-49b2-8344-9253178bd699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 09:19:48.922638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:48.914727 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:19:49.039857 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.010937 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" podUID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:49.040511 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.040474 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:19:49.081121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.081093 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:19:49.127787 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.127730 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:49.128491 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.128470 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:19:49.168257 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.168218 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 09:19:49.344056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.343944 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 09:19:49.344481 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.173246 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:49.349632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.345473 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:19:49.412622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.390345 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 09:19:49.412622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.325517 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:49.529752 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.529711 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:49.530024 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.529988 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" podUID="13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:49.672615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.671893 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 09:19:49.672615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.671966 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" podUID="929a1bc8-672b-49b2-8344-9253178bd699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 09:19:49.699284 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.699250 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 09:19:49.699680 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.699649 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 09:19:49.827616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.827454 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:49.916123 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:49.916064 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:50.069675 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.057847 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" podUID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 09:19:50.138609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.134753 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:50.367675 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.345737 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 09:19:50.367675 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.347974 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 09:19:50.367675 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.353749 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:50.415617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.403156 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:19:50.470616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.443911 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 09:19:50.539615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.463770 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:50.820494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.543403 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn"] Oct 30 09:19:50.820494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.663757 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 09:19:50.820494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.663824 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 09:19:50.820494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.664339 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 09:19:50.820494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.664379 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" podUID="929a1bc8-672b-49b2-8344-9253178bd699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 09:19:50.892648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.883755 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 09:19:50.924577 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.924540 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:19:50.928896 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.928757 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn"] Oct 30 09:19:50.932039 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.925485 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:50.949833 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.946917 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:50.984486 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.984438 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 09:19:51.254988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:50.984745 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:51.254988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.056418 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.254988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.236606 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.254988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.236667 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.254988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.236835 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.254988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.236919 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n94b9\" (UniqueName: \"kubernetes.io/projected/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-kube-api-access-n94b9\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.254988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.236989 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.254988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.237245 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.254988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.237310 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.254988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.237357 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.254988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.083114 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" podUID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 09:19:51.258224 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.142890 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 09:19:51.359769 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.359718 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 09:19:51.435095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.360115 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 09:19:51.483657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.361679 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 09:19:51.484240 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.388979 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.486085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.389650 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.486085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.484705 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.486085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.484874 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.514617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.510725 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.514617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.510840 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n94b9\" (UniqueName: \"kubernetes.io/projected/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-kube-api-access-n94b9\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.514617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.510919 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.514617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.511018 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.514617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.511067 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.514617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.511103 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.514617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.511743 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.530342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.530308 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.550210 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.531873 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.588203 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.549716 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.588452 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.566744 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:51.589020 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.571095 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:51.675017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.674980 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 09:19:51.675359 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.675327 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 09:19:51.685025 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.684971 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.763679 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.714444 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:51.765298 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.765245 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" podUID="f2b81448-0b6f-4093-9bdc-37139355b7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:51.765510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.729644 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.765698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.727740 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.870678 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.843079 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n94b9\" (UniqueName: \"kubernetes.io/projected/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-kube-api-access-n94b9\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:51.871729 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.870996 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:51.898377 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:51.897732 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:52.178388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.087617 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:19:52.178727 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.105860 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:52.179310 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.179291 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:19:52.256612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.250916 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" podUID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:52.319819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.257660 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:52.319819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.308873 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:52.327001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.326951 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:19:52.356140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.330973 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:52.356140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.351533 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:19:52.439845 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.439649 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:52.658640 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.605780 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 09:19:52.658640 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.607848 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:19:52.658640 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.616691 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 09:19:52.658640 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.623001 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:19:52.658640 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.642902 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:19:52.662630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.659197 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 09:19:52.679008 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.678963 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded" Oct 30 09:19:52.801722 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.800913 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:52.801722 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.800971 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" podUID="929a1bc8-672b-49b2-8344-9253178bd699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:52.805868 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.803664 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:19:52.818390 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.818362 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:19:52.860760 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.860690 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn"] Oct 30 09:19:52.915742 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.915679 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:19:52.916203 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.916118 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:19:52.916203 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:52.916176 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:19:53.512614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:53.512564 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" event={"ID":"b4b3c5d0-f0a6-4077-b212-724b6dbe378a","Type":"ContainerStarted","Data":"b68000275cda1ad5b64c3c52f663266e22cd1527368e29f1f1c9c141d68dae17"} Oct 30 09:19:53.512874 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:53.512857 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" event={"ID":"b4b3c5d0-f0a6-4077-b212-724b6dbe378a","Type":"ContainerStarted","Data":"09ec4d634d09f21f02b8dcfc6db94b439d4d24d1f69943bb2a2795881a2ecda6"} Oct 30 09:19:53.570424 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:53.569728 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" podStartSLOduration=4.569712509 podStartE2EDuration="4.569712509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:53.568910922 +0000 UTC m=+1511.431616815" watchObservedRunningTime="2026-10-30 09:19:53.569712509 +0000 UTC m=+1511.432418387" Oct 30 09:19:53.627166 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:53.627126 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:53.627344 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:53.627209 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:53.846674 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:53.842791 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:53.846674 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:53.842889 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" podUID="929a1bc8-672b-49b2-8344-9253178bd699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:54.106430 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:54.106334 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:19:54.108140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:54.108112 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 09:19:54.108271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:54.108167 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" podUID="b4b3c5d0-f0a6-4077-b212-724b6dbe378a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 09:19:54.563609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:54.558069 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:54.563609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:54.560508 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:54.572614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:54.571006 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:19:54.672799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:54.672762 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:54.672972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:54.672825 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:55.057566 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:55.057518 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:55.058033 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:55.058008 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:55.106306 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:55.106268 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 09:19:55.106492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:55.106333 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" podUID="b4b3c5d0-f0a6-4077-b212-724b6dbe378a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 09:19:55.149702 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:55.139220 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:19:56.106917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:56.106875 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 09:19:56.107337 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:56.106952 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" podUID="b4b3c5d0-f0a6-4077-b212-724b6dbe378a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 09:19:57.106989 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:57.106951 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 09:19:57.107506 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:57.107009 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" podUID="b4b3c5d0-f0a6-4077-b212-724b6dbe378a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 09:19:58.107300 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:58.107266 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 09:19:58.107817 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:58.107335 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" podUID="b4b3c5d0-f0a6-4077-b212-724b6dbe378a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 09:19:59.106914 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:59.106877 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 09:19:59.107125 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:19:59.106949 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" podUID="b4b3c5d0-f0a6-4077-b212-724b6dbe378a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 09:20:00.106783 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:00.106753 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 09:20:00.107187 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:00.106804 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" podUID="b4b3c5d0-f0a6-4077-b212-724b6dbe378a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 09:20:01.111175 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:01.111143 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:20:01.111629 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:01.111610 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:20:01.112600 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:01.112566 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:20:06.278761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.278728 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95"] Oct 30 09:20:06.283172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.283143 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.315187 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.315151 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95"] Oct 30 09:20:06.373875 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.373841 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjhzg\" (UniqueName: \"kubernetes.io/projected/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-kube-api-access-sjhzg\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.374099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.373911 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.374099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.373940 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.374099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.373986 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.374099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.374087 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.374311 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.374173 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.374311 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.374221 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.374925 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.374899 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.377302 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.377266 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.478638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.478579 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.478856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.478710 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.478856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.478753 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjhzg\" (UniqueName: \"kubernetes.io/projected/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-kube-api-access-sjhzg\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.478856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.478797 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.478856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.478823 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.479069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.478866 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.479069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.479014 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.479174 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.479125 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.479269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.479246 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.479338 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.479301 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.479641 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.479574 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.479885 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.479864 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.480182 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.480164 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.480281 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.480259 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.481499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.481474 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.481751 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.481733 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.500287 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.500245 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.500448 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.500316 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjhzg\" (UniqueName: \"kubernetes.io/projected/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-kube-api-access-sjhzg\") pod \"maas-default-gateway-openshift-default-77f556d647-p6d95\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.602512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.602395 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:06.933972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.933942 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95"] Oct 30 09:20:06.935419 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:20:06.935391 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2176b3a_82cf_41b4_ae69_c5dea2a721f7.slice/crio-a7e2c63429b03d01c1e0ab9b2ebdb8ea466584458487906a31d38cb2c466cbdc WatchSource:0}: Error finding container a7e2c63429b03d01c1e0ab9b2ebdb8ea466584458487906a31d38cb2c466cbdc: Status 404 returned error can't find the container with id a7e2c63429b03d01c1e0ab9b2ebdb8ea466584458487906a31d38cb2c466cbdc Oct 30 09:20:06.938188 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.938151 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:20:06.938275 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.938230 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:20:06.938275 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:06.938261 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:20:07.748501 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:07.748459 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" event={"ID":"f2176b3a-82cf-41b4-ae69-c5dea2a721f7","Type":"ContainerStarted","Data":"297ec831868790705f1be4942c173e7cde72fd7981162e83b40d393d8581b068"} Oct 30 09:20:07.748995 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:07.748532 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" event={"ID":"f2176b3a-82cf-41b4-ae69-c5dea2a721f7","Type":"ContainerStarted","Data":"a7e2c63429b03d01c1e0ab9b2ebdb8ea466584458487906a31d38cb2c466cbdc"} Oct 30 09:20:07.808418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:07.808343 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" podStartSLOduration=1.808322014 podStartE2EDuration="1.808322014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:20:07.806067918 +0000 UTC m=+1525.668773739" watchObservedRunningTime="2026-10-30 09:20:07.808322014 +0000 UTC m=+1525.671027835" Oct 30 09:20:08.603447 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:08.603415 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:08.604988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:08.604964 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-p6d95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 09:20:08.605123 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:08.605019 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" podUID="f2176b3a-82cf-41b4-ae69-c5dea2a721f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 09:20:08.954671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:08.952840 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56"] Oct 30 09:20:08.958308 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:08.958279 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56"] Oct 30 09:20:08.958491 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:08.958387 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:08.988806 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:08.988772 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8eada0cf\"/\"kube-root-ca.crt\"" Oct 30 09:20:08.989031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:08.988849 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8eada0cf\"/\"openshift-service-ca.crt\"" Oct 30 09:20:08.989099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:08.989029 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8eada0cf\"/\"default-dockercfg-h8l5j\"" Oct 30 09:20:08.989151 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:08.989141 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8eada0cf\"/\"e2e-lifecycle-model-8eada0cf-kserve-self-signed-certs\"" Oct 30 09:20:09.054760 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.054726 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-model-cache\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.054964 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.054797 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz8ds\" (UniqueName: \"kubernetes.io/projected/e91e4e53-b0ac-491a-8baf-4550976139aa-kube-api-access-gz8ds\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.054964 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.054833 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-dshm\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.054964 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.054906 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91e4e53-b0ac-491a-8baf-4550976139aa-tls-certs\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.054964 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.054947 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-tmp-dir\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.055099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.055016 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-home\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.157026 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.156991 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-home\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.157220 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.157089 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-model-cache\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.157220 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.157163 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz8ds\" (UniqueName: \"kubernetes.io/projected/e91e4e53-b0ac-491a-8baf-4550976139aa-kube-api-access-gz8ds\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.157220 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.157206 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-dshm\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.157410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.157305 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91e4e53-b0ac-491a-8baf-4550976139aa-tls-certs\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.157410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.157364 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-tmp-dir\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.157506 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.157453 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-home\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.158126 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.158097 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-model-cache\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.158309 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.158287 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-tmp-dir\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.160577 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.160547 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91e4e53-b0ac-491a-8baf-4550976139aa-tls-certs\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.161518 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.161489 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-dshm\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.187829 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.187760 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz8ds\" (UniqueName: \"kubernetes.io/projected/e91e4e53-b0ac-491a-8baf-4550976139aa-kube-api-access-gz8ds\") pod \"e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.285350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.284941 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:09.603416 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.603329 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-p6d95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 09:20:09.603416 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.603391 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" podUID="f2176b3a-82cf-41b4-ae69-c5dea2a721f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 09:20:09.663081 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.663055 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56"] Oct 30 09:20:09.664649 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:20:09.664619 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode91e4e53_b0ac_491a_8baf_4550976139aa.slice/crio-aa433a9ea1088c19377f4b2e543e411753b081cfc800087bfe3cce03623db4b6 WatchSource:0}: Error finding container aa433a9ea1088c19377f4b2e543e411753b081cfc800087bfe3cce03623db4b6: Status 404 returned error can't find the container with id aa433a9ea1088c19377f4b2e543e411753b081cfc800087bfe3cce03623db4b6 Oct 30 09:20:09.776634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:09.776572 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" event={"ID":"e91e4e53-b0ac-491a-8baf-4550976139aa","Type":"ContainerStarted","Data":"aa433a9ea1088c19377f4b2e543e411753b081cfc800087bfe3cce03623db4b6"} Oct 30 09:20:10.602837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:10.602803 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-p6d95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 09:20:10.603337 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:10.602874 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" podUID="f2176b3a-82cf-41b4-ae69-c5dea2a721f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 09:20:10.785134 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:10.785098 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" event={"ID":"e91e4e53-b0ac-491a-8baf-4550976139aa","Type":"ContainerStarted","Data":"545b953d9ce56227768585815cc99a8ff75b43892e223fa711792356a712305e"} Oct 30 09:20:11.286722 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:11.286683 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:11.606143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:11.605110 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-p6d95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 09:20:11.606143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:11.605169 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" podUID="f2176b3a-82cf-41b4-ae69-c5dea2a721f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 09:20:12.602927 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:12.602894 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-p6d95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 09:20:12.603118 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:12.602946 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" podUID="f2176b3a-82cf-41b4-ae69-c5dea2a721f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 09:20:12.998957 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:12.998904 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" podStartSLOduration=4.9988904309999995 podStartE2EDuration="4.998890431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:20:10.843085397 +0000 UTC m=+1528.705791220" watchObservedRunningTime="2026-10-30 09:20:12.998890431 +0000 UTC m=+1530.861596253" Oct 30 09:20:12.999563 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:12.999541 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz"] Oct 30 09:20:13.004377 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.004351 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.045265 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.041800 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8d1106-openshift-default-dockercfg-n8hb7\"" Oct 30 09:20:13.050111 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.050049 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz"] Oct 30 09:20:13.170621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.165415 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-workload-socket\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.170621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.165528 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-envoy\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.170621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.165651 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-workload-certs\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.170621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.165682 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istiod-ca-cert\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.170621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.165760 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-credential-socket\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.170621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.165835 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh5kv\" (UniqueName: \"kubernetes.io/projected/523627cb-cbb0-4c8e-8783-5fe8203a57c3-kube-api-access-qh5kv\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.170621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.165933 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-data\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.170621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.165984 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-token\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.170621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.166046 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-podinfo\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.269736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.267829 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-credential-socket\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.269736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.267928 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh5kv\" (UniqueName: \"kubernetes.io/projected/523627cb-cbb0-4c8e-8783-5fe8203a57c3-kube-api-access-qh5kv\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.269736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.268015 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-data\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.269736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.268062 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-token\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.269736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.268113 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-podinfo\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.269736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.268217 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-workload-socket\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.269736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.268295 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-envoy\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.269736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.268360 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-workload-certs\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.269736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.268386 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istiod-ca-cert\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.269736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.269279 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istiod-ca-cert\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.269736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.269574 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-credential-socket\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.270438 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.270187 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-data\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.271205 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.271172 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-workload-certs\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.282615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.271561 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-workload-socket\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.295725 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.295694 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-envoy\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.304160 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.304125 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-token\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.325671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.318234 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-podinfo\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.327089 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.327016 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh5kv\" (UniqueName: \"kubernetes.io/projected/523627cb-cbb0-4c8e-8783-5fe8203a57c3-kube-api-access-qh5kv\") pod \"e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.350114 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.349876 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:13.603092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.602990 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-p6d95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 09:20:13.603092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.603054 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" podUID="f2176b3a-82cf-41b4-ae69-c5dea2a721f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 09:20:13.712838 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.712809 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz"] Oct 30 09:20:13.715631 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:20:13.715555 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523627cb_cbb0_4c8e_8783_5fe8203a57c3.slice/crio-c93434c0590775e341f4cce14846eb7838833889a3e0e0127f99be71791389f5 WatchSource:0}: Error finding container c93434c0590775e341f4cce14846eb7838833889a3e0e0127f99be71791389f5: Status 404 returned error can't find the container with id c93434c0590775e341f4cce14846eb7838833889a3e0e0127f99be71791389f5 Oct 30 09:20:13.718415 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.718378 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:20:13.718523 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.718480 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:20:13.718578 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.718524 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:20:13.809037 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:13.808995 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" event={"ID":"523627cb-cbb0-4c8e-8783-5fe8203a57c3","Type":"ContainerStarted","Data":"c93434c0590775e341f4cce14846eb7838833889a3e0e0127f99be71791389f5"} Oct 30 09:20:14.603087 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:14.603049 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-p6d95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 09:20:14.603507 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:14.603111 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" podUID="f2176b3a-82cf-41b4-ae69-c5dea2a721f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 09:20:14.818403 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:14.818362 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" event={"ID":"523627cb-cbb0-4c8e-8783-5fe8203a57c3","Type":"ContainerStarted","Data":"cf8c1adf1a5559733e7bfc0e89850c25d2d752e34f46cb256b78c54c5c8d8b23"} Oct 30 09:20:14.879816 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:14.879754 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" podStartSLOduration=2.879737477 podStartE2EDuration="2.879737477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:20:14.878180329 +0000 UTC m=+1532.740886150" watchObservedRunningTime="2026-10-30 09:20:14.879737477 +0000 UTC m=+1532.742443299" Oct 30 09:20:15.352419 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:15.352324 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:15.476965 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:15.476931 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:15.815753 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:20:15.815701 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:20:15.826238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:15.826196 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:15.827291 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:15.827271 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:16.180449 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:16.180416 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:16.180788 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:16.180767 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:16.181754 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:16.181735 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:20:19.286335 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:19.286292 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:19.302940 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:19.302906 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:19.863861 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:19.863825 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:20:21.341292 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.341248 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx"] Oct 30 09:20:21.346664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.346631 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.387465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.387428 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx"] Oct 30 09:20:21.431144 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.431104 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d11017ce-5d73-4775-a489-463a118338dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.431351 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.431284 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.431441 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.431418 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.431497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.431471 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d11017ce-5d73-4775-a489-463a118338dd-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.431557 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.431542 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.431687 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.431671 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d11017ce-5d73-4775-a489-463a118338dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.431746 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.431708 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.431799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.431741 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp499\" (UniqueName: \"kubernetes.io/projected/d11017ce-5d73-4775-a489-463a118338dd-kube-api-access-lp499\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.431799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.431793 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.533626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.533547 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d11017ce-5d73-4775-a489-463a118338dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.533626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.533621 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.533887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.533656 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp499\" (UniqueName: \"kubernetes.io/projected/d11017ce-5d73-4775-a489-463a118338dd-kube-api-access-lp499\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.533887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.533700 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.533887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.533748 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d11017ce-5d73-4775-a489-463a118338dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.533887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.533833 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.534094 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.533935 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.534094 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.533979 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d11017ce-5d73-4775-a489-463a118338dd-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.534094 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.534058 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.534232 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.534179 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.534352 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.534329 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d11017ce-5d73-4775-a489-463a118338dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.534615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.534570 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.534738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.534718 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.534935 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.534913 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.536346 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.536318 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.536562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.536539 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d11017ce-5d73-4775-a489-463a118338dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.558005 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.557918 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp499\" (UniqueName: \"kubernetes.io/projected/d11017ce-5d73-4775-a489-463a118338dd-kube-api-access-lp499\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.561346 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.561271 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d11017ce-5d73-4775-a489-463a118338dd-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-t2zhx\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.665122 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.665085 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:21.984850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.984820 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx"] Oct 30 09:20:21.985470 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:20:21.985438 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11017ce_5d73_4775_a489_463a118338dd.slice/crio-f82f9034a86f3b520e45261a14832954e8145a60995b2a7e979e83509d86ca98 WatchSource:0}: Error finding container f82f9034a86f3b520e45261a14832954e8145a60995b2a7e979e83509d86ca98: Status 404 returned error can't find the container with id f82f9034a86f3b520e45261a14832954e8145a60995b2a7e979e83509d86ca98 Oct 30 09:20:21.988393 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.988354 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:20:21.988497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.988459 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:20:21.988575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:21.988520 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:20:22.874280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:22.874233 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" event={"ID":"d11017ce-5d73-4775-a489-463a118338dd","Type":"ContainerStarted","Data":"c5a985f77dbd5fc0f9fab9fb7f2992f6a9886c57f2598586e52ed030eacafe22"} Oct 30 09:20:22.874792 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:22.874302 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" event={"ID":"d11017ce-5d73-4775-a489-463a118338dd","Type":"ContainerStarted","Data":"f82f9034a86f3b520e45261a14832954e8145a60995b2a7e979e83509d86ca98"} Oct 30 09:20:22.953333 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:22.953277 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" podStartSLOduration=1.9532636719999998 podStartE2EDuration="1.953263672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:20:22.937136342 +0000 UTC m=+1540.799842197" watchObservedRunningTime="2026-10-30 09:20:22.953263672 +0000 UTC m=+1540.815969527" Oct 30 09:20:23.666053 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:23.666013 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:23.667681 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:23.667655 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t2zhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 09:20:23.667834 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:23.667703 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" podUID="d11017ce-5d73-4775-a489-463a118338dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 09:20:24.666337 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:24.666301 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t2zhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 09:20:24.666809 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:24.666364 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" podUID="d11017ce-5d73-4775-a489-463a118338dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 09:20:25.666822 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:25.666785 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t2zhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 09:20:25.667231 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:25.666890 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" podUID="d11017ce-5d73-4775-a489-463a118338dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 09:20:26.671625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:26.671028 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t2zhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 09:20:26.671625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:26.671096 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" podUID="d11017ce-5d73-4775-a489-463a118338dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 09:20:27.666463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:27.666431 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t2zhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 09:20:27.666736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:27.666488 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" podUID="d11017ce-5d73-4775-a489-463a118338dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 09:20:28.666484 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:28.666449 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t2zhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 09:20:28.667047 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:28.666521 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" podUID="d11017ce-5d73-4775-a489-463a118338dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 09:20:29.665741 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:29.665708 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t2zhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 09:20:29.666016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:29.665764 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" podUID="d11017ce-5d73-4775-a489-463a118338dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 09:20:30.813082 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:30.813044 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:30.813605 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:30.813576 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:30.814442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:30.814415 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:20:33.731037 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:33.730995 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6"] Oct 30 09:20:33.735459 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:33.735433 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:20:33.754812 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:33.754726 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6"] Oct 30 09:20:33.769612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:33.768609 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdab4295-ded3-42f4-859a-7bc26ffc0f40-openshift-service-ca\") pod \"maas-api-e2e-api-b-8d1106-988d9844c-qp8h6\" (UID: \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:20:33.769612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:33.768748 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwnnx\" (UniqueName: \"kubernetes.io/projected/cdab4295-ded3-42f4-859a-7bc26ffc0f40-kube-api-access-hwnnx\") pod \"maas-api-e2e-api-b-8d1106-988d9844c-qp8h6\" (UID: \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:20:33.769612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:33.768987 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdab4295-ded3-42f4-859a-7bc26ffc0f40-maas-api-tls\") pod \"maas-api-e2e-api-b-8d1106-988d9844c-qp8h6\" (UID: \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:20:33.769612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:33.769126 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdab4295-ded3-42f4-859a-7bc26ffc0f40-metrics-certs\") pod \"maas-api-e2e-api-b-8d1106-988d9844c-qp8h6\" (UID: \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:20:33.786034 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:33.785980 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8d1106-serving-cert\"" Oct 30 09:20:33.870705 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:33.870655 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdab4295-ded3-42f4-859a-7bc26ffc0f40-maas-api-tls\") pod \"maas-api-e2e-api-b-8d1106-988d9844c-qp8h6\" (UID: \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:20:33.870903 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:33.870768 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdab4295-ded3-42f4-859a-7bc26ffc0f40-metrics-certs\") pod \"maas-api-e2e-api-b-8d1106-988d9844c-qp8h6\" (UID: \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:20:33.870903 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:33.870869 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdab4295-ded3-42f4-859a-7bc26ffc0f40-openshift-service-ca\") pod \"maas-api-e2e-api-b-8d1106-988d9844c-qp8h6\" (UID: \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:20:33.871006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:33.870948 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwnnx\" (UniqueName: \"kubernetes.io/projected/cdab4295-ded3-42f4-859a-7bc26ffc0f40-kube-api-access-hwnnx\") pod \"maas-api-e2e-api-b-8d1106-988d9844c-qp8h6\" (UID: \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:20:33.871529 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:33.871504 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdab4295-ded3-42f4-859a-7bc26ffc0f40-openshift-service-ca\") pod \"maas-api-e2e-api-b-8d1106-988d9844c-qp8h6\" (UID: \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:20:33.873366 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:33.873343 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdab4295-ded3-42f4-859a-7bc26ffc0f40-metrics-certs\") pod \"maas-api-e2e-api-b-8d1106-988d9844c-qp8h6\" (UID: \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:20:33.873474 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:33.873382 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdab4295-ded3-42f4-859a-7bc26ffc0f40-maas-api-tls\") pod \"maas-api-e2e-api-b-8d1106-988d9844c-qp8h6\" (UID: \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:20:33.891675 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:33.891637 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwnnx\" (UniqueName: \"kubernetes.io/projected/cdab4295-ded3-42f4-859a-7bc26ffc0f40-kube-api-access-hwnnx\") pod \"maas-api-e2e-api-b-8d1106-988d9844c-qp8h6\" (UID: \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:20:34.051259 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:34.051151 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:20:34.358737 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:34.358709 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6"] Oct 30 09:20:34.360270 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:20:34.360244 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdab4295_ded3_42f4_859a_7bc26ffc0f40.slice/crio-d79fd6620a1a3f28a05859253ad1cacbd6caf1239231b935e316364704aaa35f WatchSource:0}: Error finding container d79fd6620a1a3f28a05859253ad1cacbd6caf1239231b935e316364704aaa35f: Status 404 returned error can't find the container with id d79fd6620a1a3f28a05859253ad1cacbd6caf1239231b935e316364704aaa35f Oct 30 09:20:34.362098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:34.362076 5938 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:20:34.962669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:34.962623 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" event={"ID":"cdab4295-ded3-42f4-859a-7bc26ffc0f40","Type":"ContainerStarted","Data":"b89a844d1b0fe8740b0255fe3723c1f29721cab77f2844806cba9f02932adbf4"} Oct 30 09:20:34.963143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:34.962690 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" event={"ID":"cdab4295-ded3-42f4-859a-7bc26ffc0f40","Type":"ContainerStarted","Data":"d79fd6620a1a3f28a05859253ad1cacbd6caf1239231b935e316364704aaa35f"} Oct 30 09:20:34.963143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:34.962712 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:20:35.009255 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:35.009190 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" podStartSLOduration=1.537292215 podStartE2EDuration="2.009168466s" totalImagesPullingTime="471.876251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:20:35.004572198 +0000 UTC m=+1552.867278023" watchObservedRunningTime="2026-10-30 09:20:35.009168466 +0000 UTC m=+1552.871874288" Oct 30 09:20:40.975806 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:40.975774 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:20:45.811491 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:20:45.811446 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:20:58.201425 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:58.201324 5938 generic.go:356] "Generic (PLEG): container finished" podID="523627cb-cbb0-4c8e-8783-5fe8203a57c3" containerID="cf8c1adf1a5559733e7bfc0e89850c25d2d752e34f46cb256b78c54c5c8d8b23" exitCode=0 Oct 30 09:20:58.201425 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:58.201406 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" event={"ID":"523627cb-cbb0-4c8e-8783-5fe8203a57c3","Type":"ContainerDied","Data":"cf8c1adf1a5559733e7bfc0e89850c25d2d752e34f46cb256b78c54c5c8d8b23"} Oct 30 09:20:58.202006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:58.201936 5938 scope.go:122] "RemoveContainer" containerID="cf8c1adf1a5559733e7bfc0e89850c25d2d752e34f46cb256b78c54c5c8d8b23" Oct 30 09:20:58.203582 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:58.203545 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:20:58.203715 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:58.203644 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:20:58.203715 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:58.203682 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:20:59.211095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:59.211060 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" event={"ID":"523627cb-cbb0-4c8e-8783-5fe8203a57c3","Type":"ContainerStarted","Data":"94e093418121d5e79d5bee5710c4d5ee2f05ea5eab8fa1ebdaecb15dffee878e"} Oct 30 09:20:59.352180 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:59.352142 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:20:59.354053 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:59.354016 5938 patch_prober.go:29] interesting pod/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:20:59.354358 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:20:59.354328 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" podUID="523627cb-cbb0-4c8e-8783-5fe8203a57c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 09:21:00.351829 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:21:00.351788 5938 patch_prober.go:29] interesting pod/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:21:00.352272 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:21:00.351873 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" podUID="523627cb-cbb0-4c8e-8783-5fe8203a57c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 09:21:00.840616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:21:00.839703 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:21:01.351557 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:21:01.351513 5938 patch_prober.go:29] interesting pod/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:21:01.351808 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:21:01.351579 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" podUID="523627cb-cbb0-4c8e-8783-5fe8203a57c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 09:21:02.359065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:21:02.359024 5938 patch_prober.go:29] interesting pod/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:21:02.359723 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:21:02.359686 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" podUID="523627cb-cbb0-4c8e-8783-5fe8203a57c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 09:21:03.352278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:21:03.352241 5938 patch_prober.go:29] interesting pod/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:21:03.352472 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:21:03.352300 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" podUID="523627cb-cbb0-4c8e-8783-5fe8203a57c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 09:21:04.352415 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:21:04.352030 5938 patch_prober.go:29] interesting pod/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:21:04.352415 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:21:04.352097 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" podUID="523627cb-cbb0-4c8e-8783-5fe8203a57c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 09:21:05.356719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:21:05.356688 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:21:05.358004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:21:05.357980 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:21:15.811425 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:21:15.811389 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:21:45.812702 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:21:45.812663 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:22:03.407203 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:03.407159 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c"] Oct 30 09:22:03.407751 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:03.407458 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" podUID="dc0c0552-4af6-429f-bda5-bf438f8703a6" containerName="maas-api" containerID="cri-o://860a9bd4c21ad5c66d1cb4617a25c229506e64e40a77febf031c6aad2745e5c0" gracePeriod=30 Oct 30 09:22:03.886128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:03.885809 5938 generic.go:356] "Generic (PLEG): container finished" podID="dc0c0552-4af6-429f-bda5-bf438f8703a6" containerID="860a9bd4c21ad5c66d1cb4617a25c229506e64e40a77febf031c6aad2745e5c0" exitCode=0 Oct 30 09:22:03.886128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:03.886051 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" event={"ID":"dc0c0552-4af6-429f-bda5-bf438f8703a6","Type":"ContainerDied","Data":"860a9bd4c21ad5c66d1cb4617a25c229506e64e40a77febf031c6aad2745e5c0"} Oct 30 09:22:04.904618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:04.893472 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:22:04.904618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:04.898746 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" event={"ID":"dc0c0552-4af6-429f-bda5-bf438f8703a6","Type":"ContainerDied","Data":"e5d1c1cac14645806ff4c79cc7774c04ed6d7151fc5116dccfa194304170ee21"} Oct 30 09:22:04.904618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:04.898823 5938 scope.go:122] "RemoveContainer" containerID="860a9bd4c21ad5c66d1cb4617a25c229506e64e40a77febf031c6aad2745e5c0" Oct 30 09:22:04.904618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:04.899005 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c" Oct 30 09:22:04.981825 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:04.981768 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc0c0552-4af6-429f-bda5-bf438f8703a6-kube-api-access-cs5wv\" (UniqueName: \"kubernetes.io/projected/dc0c0552-4af6-429f-bda5-bf438f8703a6-kube-api-access-cs5wv\") pod \"dc0c0552-4af6-429f-bda5-bf438f8703a6\" (UID: \"dc0c0552-4af6-429f-bda5-bf438f8703a6\") " Oct 30 09:22:04.982037 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:04.981893 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc0c0552-4af6-429f-bda5-bf438f8703a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc0c0552-4af6-429f-bda5-bf438f8703a6-maas-api-tls\") pod \"dc0c0552-4af6-429f-bda5-bf438f8703a6\" (UID: \"dc0c0552-4af6-429f-bda5-bf438f8703a6\") " Oct 30 09:22:04.982037 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:04.981964 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc0c0552-4af6-429f-bda5-bf438f8703a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc0c0552-4af6-429f-bda5-bf438f8703a6-openshift-service-ca\") pod \"dc0c0552-4af6-429f-bda5-bf438f8703a6\" (UID: \"dc0c0552-4af6-429f-bda5-bf438f8703a6\") " Oct 30 09:22:04.982168 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:04.982085 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc0c0552-4af6-429f-bda5-bf438f8703a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc0c0552-4af6-429f-bda5-bf438f8703a6-metrics-certs\") pod \"dc0c0552-4af6-429f-bda5-bf438f8703a6\" (UID: \"dc0c0552-4af6-429f-bda5-bf438f8703a6\") " Oct 30 09:22:04.982674 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:04.982643 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc0c0552-4af6-429f-bda5-bf438f8703a6-openshift-service-ca" pod "dc0c0552-4af6-429f-bda5-bf438f8703a6" (UID: "dc0c0552-4af6-429f-bda5-bf438f8703a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:22:04.996655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:04.985366 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc0c0552-4af6-429f-bda5-bf438f8703a6-maas-api-tls" pod "dc0c0552-4af6-429f-bda5-bf438f8703a6" (UID: "dc0c0552-4af6-429f-bda5-bf438f8703a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:22:04.996655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:04.985683 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0c0552-4af6-429f-bda5-bf438f8703a6-kube-api-access-cs5wv" pod "dc0c0552-4af6-429f-bda5-bf438f8703a6" (UID: "dc0c0552-4af6-429f-bda5-bf438f8703a6"). InnerVolumeSpecName "kube-api-access-cs5wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:22:04.996655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:04.986005 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc0c0552-4af6-429f-bda5-bf438f8703a6-metrics-certs" pod "dc0c0552-4af6-429f-bda5-bf438f8703a6" (UID: "dc0c0552-4af6-429f-bda5-bf438f8703a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:22:05.084187 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:05.084089 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs5wv\" (UniqueName: \"kubernetes.io/projected/dc0c0552-4af6-429f-bda5-bf438f8703a6-kube-api-access-cs5wv\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:05.084187 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:05.084126 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc0c0552-4af6-429f-bda5-bf438f8703a6-maas-api-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:05.084187 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:05.084138 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc0c0552-4af6-429f-bda5-bf438f8703a6-openshift-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:05.084187 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:05.084147 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc0c0552-4af6-429f-bda5-bf438f8703a6-metrics-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:05.248504 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:05.248469 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c"] Oct 30 09:22:05.251490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:05.251455 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-fgh2c"] Oct 30 09:22:06.258626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:06.258565 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc0c0552-4af6-429f-bda5-bf438f8703a6" path="/var/lib/kubelet/pods/dc0c0552-4af6-429f-bda5-bf438f8703a6/volumes" Oct 30 09:22:15.811086 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:22:15.811045 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:22:18.426038 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:18.415964 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6"] Oct 30 09:22:18.426038 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:18.416218 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" podUID="cdab4295-ded3-42f4-859a-7bc26ffc0f40" containerName="maas-api" containerID="cri-o://b89a844d1b0fe8740b0255fe3723c1f29721cab77f2844806cba9f02932adbf4" gracePeriod=30 Oct 30 09:22:18.898611 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:18.898011 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:22:18.967416 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:18.967315 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdab4295-ded3-42f4-859a-7bc26ffc0f40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdab4295-ded3-42f4-859a-7bc26ffc0f40-maas-api-tls\") pod \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\" (UID: \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\") " Oct 30 09:22:18.967416 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:18.967410 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdab4295-ded3-42f4-859a-7bc26ffc0f40-kube-api-access-hwnnx\" (UniqueName: \"kubernetes.io/projected/cdab4295-ded3-42f4-859a-7bc26ffc0f40-kube-api-access-hwnnx\") pod \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\" (UID: \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\") " Oct 30 09:22:18.967690 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:18.967487 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdab4295-ded3-42f4-859a-7bc26ffc0f40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdab4295-ded3-42f4-859a-7bc26ffc0f40-metrics-certs\") pod \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\" (UID: \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\") " Oct 30 09:22:18.967690 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:18.967551 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdab4295-ded3-42f4-859a-7bc26ffc0f40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdab4295-ded3-42f4-859a-7bc26ffc0f40-openshift-service-ca\") pod \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\" (UID: \"cdab4295-ded3-42f4-859a-7bc26ffc0f40\") " Oct 30 09:22:18.968047 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:18.968015 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdab4295-ded3-42f4-859a-7bc26ffc0f40-openshift-service-ca" pod "cdab4295-ded3-42f4-859a-7bc26ffc0f40" (UID: "cdab4295-ded3-42f4-859a-7bc26ffc0f40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:22:18.968474 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:18.968451 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdab4295-ded3-42f4-859a-7bc26ffc0f40-openshift-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:18.970813 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:18.970724 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdab4295-ded3-42f4-859a-7bc26ffc0f40-maas-api-tls" pod "cdab4295-ded3-42f4-859a-7bc26ffc0f40" (UID: "cdab4295-ded3-42f4-859a-7bc26ffc0f40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:22:18.978830 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:18.977060 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdab4295-ded3-42f4-859a-7bc26ffc0f40-kube-api-access-hwnnx" pod "cdab4295-ded3-42f4-859a-7bc26ffc0f40" (UID: "cdab4295-ded3-42f4-859a-7bc26ffc0f40"). InnerVolumeSpecName "kube-api-access-hwnnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:22:19.010905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:19.006726 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdab4295-ded3-42f4-859a-7bc26ffc0f40-metrics-certs" pod "cdab4295-ded3-42f4-859a-7bc26ffc0f40" (UID: "cdab4295-ded3-42f4-859a-7bc26ffc0f40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:22:19.025938 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:19.025890 5938 generic.go:356] "Generic (PLEG): container finished" podID="cdab4295-ded3-42f4-859a-7bc26ffc0f40" containerID="b89a844d1b0fe8740b0255fe3723c1f29721cab77f2844806cba9f02932adbf4" exitCode=0 Oct 30 09:22:19.026223 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:19.026149 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" event={"ID":"cdab4295-ded3-42f4-859a-7bc26ffc0f40","Type":"ContainerDied","Data":"b89a844d1b0fe8740b0255fe3723c1f29721cab77f2844806cba9f02932adbf4"} Oct 30 09:22:19.026314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:19.026224 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" event={"ID":"cdab4295-ded3-42f4-859a-7bc26ffc0f40","Type":"ContainerDied","Data":"d79fd6620a1a3f28a05859253ad1cacbd6caf1239231b935e316364704aaa35f"} Oct 30 09:22:19.026314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:19.026255 5938 scope.go:122] "RemoveContainer" containerID="b89a844d1b0fe8740b0255fe3723c1f29721cab77f2844806cba9f02932adbf4" Oct 30 09:22:19.026314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:19.026282 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6" Oct 30 09:22:19.038093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:19.038050 5938 scope.go:122] "RemoveContainer" containerID="b89a844d1b0fe8740b0255fe3723c1f29721cab77f2844806cba9f02932adbf4" Oct 30 09:22:19.042237 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:22:19.038630 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89a844d1b0fe8740b0255fe3723c1f29721cab77f2844806cba9f02932adbf4\": container with ID starting with b89a844d1b0fe8740b0255fe3723c1f29721cab77f2844806cba9f02932adbf4 not found: ID does not exist" containerID="b89a844d1b0fe8740b0255fe3723c1f29721cab77f2844806cba9f02932adbf4" Oct 30 09:22:19.042237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:19.038676 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89a844d1b0fe8740b0255fe3723c1f29721cab77f2844806cba9f02932adbf4"} err="failed to get container status \"b89a844d1b0fe8740b0255fe3723c1f29721cab77f2844806cba9f02932adbf4\": rpc error: code = NotFound desc = could not find container \"b89a844d1b0fe8740b0255fe3723c1f29721cab77f2844806cba9f02932adbf4\": container with ID starting with b89a844d1b0fe8740b0255fe3723c1f29721cab77f2844806cba9f02932adbf4 not found: ID does not exist" Oct 30 09:22:19.083120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:19.070908 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdab4295-ded3-42f4-859a-7bc26ffc0f40-maas-api-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:19.083120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:19.079677 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwnnx\" (UniqueName: \"kubernetes.io/projected/cdab4295-ded3-42f4-859a-7bc26ffc0f40-kube-api-access-hwnnx\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:19.083120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:19.079713 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdab4295-ded3-42f4-859a-7bc26ffc0f40-metrics-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:19.086814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:19.086767 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6"] Oct 30 09:22:19.090681 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:19.090648 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8d1106-988d9844c-qp8h6"] Oct 30 09:22:20.257715 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:20.257683 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdab4295-ded3-42f4-859a-7bc26ffc0f40" path="/var/lib/kubelet/pods/cdab4295-ded3-42f4-859a-7bc26ffc0f40/volumes" Oct 30 09:22:26.347646 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:26.347575 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz"] Oct 30 09:22:27.103958 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:27.103915 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" podUID="523627cb-cbb0-4c8e-8783-5fe8203a57c3" containerName="istio-proxy" containerID="cri-o://94e093418121d5e79d5bee5710c4d5ee2f05ea5eab8fa1ebdaecb15dffee878e" gracePeriod=30 Oct 30 09:22:32.025652 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.025580 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g"] Oct 30 09:22:32.026291 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.026272 5938 cpu_manager.go:414] "containerMap: removing container" podUID="dc0c0552-4af6-429f-bda5-bf438f8703a6" containerName="maas-api" Oct 30 09:22:32.026291 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.026292 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc0c0552-4af6-429f-bda5-bf438f8703a6" containerName="maas-api" Oct 30 09:22:32.026476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.026315 5938 cpu_manager.go:414] "containerMap: removing container" podUID="cdab4295-ded3-42f4-859a-7bc26ffc0f40" containerName="maas-api" Oct 30 09:22:32.026476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.026320 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdab4295-ded3-42f4-859a-7bc26ffc0f40" containerName="maas-api" Oct 30 09:22:32.026476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.026459 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc0c0552-4af6-429f-bda5-bf438f8703a6" containerName="maas-api" Oct 30 09:22:32.026476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.026473 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdab4295-ded3-42f4-859a-7bc26ffc0f40" containerName="maas-api" Oct 30 09:22:32.030214 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.030184 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.055906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.055869 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g"] Oct 30 09:22:32.065822 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.065771 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 09:22:32.066044 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.065831 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h44vk\"" Oct 30 09:22:32.066044 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.065870 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 09:22:32.066044 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.065898 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 09:22:32.122054 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.122022 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.122230 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.122132 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.122230 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.122188 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.122352 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.122282 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.122352 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.122331 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.122462 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.122444 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dksv\" (UniqueName: \"kubernetes.io/projected/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-kube-api-access-2dksv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.122515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.122470 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.224237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.223980 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.224237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.224017 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.224237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.224053 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.224237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.224091 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.224237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.224139 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.224237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.224224 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dksv\" (UniqueName: \"kubernetes.io/projected/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-kube-api-access-2dksv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.224575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.224256 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.224888 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.224857 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.226847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.226816 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.227077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.227055 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.227147 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.227064 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.227210 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.227185 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.246061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.246020 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.246255 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.246157 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dksv\" (UniqueName: \"kubernetes.io/projected/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-kube-api-access-2dksv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fw98g\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.344215 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.344180 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:32.392203 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.392166 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:22:32.432654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.427365 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istiod-ca-cert\") pod \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " Oct 30 09:22:32.432654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.427456 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-data\") pod \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " Oct 30 09:22:32.432654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.427510 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-envoy\") pod \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " Oct 30 09:22:32.432654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.427640 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523627cb-cbb0-4c8e-8783-5fe8203a57c3-kube-api-access-qh5kv\" (UniqueName: \"kubernetes.io/projected/523627cb-cbb0-4c8e-8783-5fe8203a57c3-kube-api-access-qh5kv\") pod \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " Oct 30 09:22:32.432654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.427727 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-workload-certs\") pod \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " Oct 30 09:22:32.432654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.427814 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-credential-socket\") pod \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " Oct 30 09:22:32.432654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.427861 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-podinfo\") pod \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " Oct 30 09:22:32.432654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.427927 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-token\" (UniqueName: \"kubernetes.io/projected/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-token\") pod \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " Oct 30 09:22:32.432654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.427984 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-workload-socket\") pod \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\" (UID: \"523627cb-cbb0-4c8e-8783-5fe8203a57c3\") " Oct 30 09:22:32.432654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.428988 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-workload-socket" pod "523627cb-cbb0-4c8e-8783-5fe8203a57c3" (UID: "523627cb-cbb0-4c8e-8783-5fe8203a57c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:22:32.432654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.429366 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istiod-ca-cert" pod "523627cb-cbb0-4c8e-8783-5fe8203a57c3" (UID: "523627cb-cbb0-4c8e-8783-5fe8203a57c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:22:32.432654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.429666 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-data" pod "523627cb-cbb0-4c8e-8783-5fe8203a57c3" (UID: "523627cb-cbb0-4c8e-8783-5fe8203a57c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:22:32.433429 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.432669 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-credential-socket" pod "523627cb-cbb0-4c8e-8783-5fe8203a57c3" (UID: "523627cb-cbb0-4c8e-8783-5fe8203a57c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:22:32.434549 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.434501 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-envoy" pod "523627cb-cbb0-4c8e-8783-5fe8203a57c3" (UID: "523627cb-cbb0-4c8e-8783-5fe8203a57c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:22:32.434852 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.434804 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-workload-certs" pod "523627cb-cbb0-4c8e-8783-5fe8203a57c3" (UID: "523627cb-cbb0-4c8e-8783-5fe8203a57c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:22:32.440245 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.439599 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-token" pod "523627cb-cbb0-4c8e-8783-5fe8203a57c3" (UID: "523627cb-cbb0-4c8e-8783-5fe8203a57c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:22:32.440773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.440743 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-podinfo" pod "523627cb-cbb0-4c8e-8783-5fe8203a57c3" (UID: "523627cb-cbb0-4c8e-8783-5fe8203a57c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:22:32.443664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.442287 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523627cb-cbb0-4c8e-8783-5fe8203a57c3-kube-api-access-qh5kv" pod "523627cb-cbb0-4c8e-8783-5fe8203a57c3" (UID: "523627cb-cbb0-4c8e-8783-5fe8203a57c3"). InnerVolumeSpecName "kube-api-access-qh5kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:22:32.530080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.530043 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:32.530080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.530075 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:32.530080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.530086 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:32.530080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.530095 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:32.530373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.530102 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh5kv\" (UniqueName: \"kubernetes.io/projected/523627cb-cbb0-4c8e-8783-5fe8203a57c3-kube-api-access-qh5kv\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:32.530373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.530113 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:32.530373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.530122 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523627cb-cbb0-4c8e-8783-5fe8203a57c3-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:32.530373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.530131 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:32.530373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.530138 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523627cb-cbb0-4c8e-8783-5fe8203a57c3-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:22:32.676237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:32.676209 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g"] Oct 30 09:22:32.678634 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:22:32.678604 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0daeb10f_9b40_4ecb_bccd_d03c06be57c7.slice/crio-270fed02e4f4b8fffef655e4077e21e8ea06045db780646b51e132e4cebf404f WatchSource:0}: Error finding container 270fed02e4f4b8fffef655e4077e21e8ea06045db780646b51e132e4cebf404f: Status 404 returned error can't find the container with id 270fed02e4f4b8fffef655e4077e21e8ea06045db780646b51e132e4cebf404f Oct 30 09:22:33.251675 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:33.249655 5938 generic.go:356] "Generic (PLEG): container finished" podID="523627cb-cbb0-4c8e-8783-5fe8203a57c3" containerID="94e093418121d5e79d5bee5710c4d5ee2f05ea5eab8fa1ebdaecb15dffee878e" exitCode=0 Oct 30 09:22:33.251675 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:33.249854 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" event={"ID":"523627cb-cbb0-4c8e-8783-5fe8203a57c3","Type":"ContainerDied","Data":"94e093418121d5e79d5bee5710c4d5ee2f05ea5eab8fa1ebdaecb15dffee878e"} Oct 30 09:22:33.251675 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:33.250072 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" event={"ID":"523627cb-cbb0-4c8e-8783-5fe8203a57c3","Type":"ContainerDied","Data":"c93434c0590775e341f4cce14846eb7838833889a3e0e0127f99be71791389f5"} Oct 30 09:22:33.251675 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:33.250112 5938 scope.go:122] "RemoveContainer" containerID="94e093418121d5e79d5bee5710c4d5ee2f05ea5eab8fa1ebdaecb15dffee878e" Oct 30 09:22:33.251675 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:33.250299 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz" Oct 30 09:22:33.268604 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:33.268542 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" event={"ID":"0daeb10f-9b40-4ecb-bccd-d03c06be57c7","Type":"ContainerStarted","Data":"270fed02e4f4b8fffef655e4077e21e8ea06045db780646b51e132e4cebf404f"} Oct 30 09:22:33.275669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:33.275371 5938 scope.go:122] "RemoveContainer" containerID="cf8c1adf1a5559733e7bfc0e89850c25d2d752e34f46cb256b78c54c5c8d8b23" Oct 30 09:22:33.289138 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:33.289115 5938 scope.go:122] "RemoveContainer" containerID="94e093418121d5e79d5bee5710c4d5ee2f05ea5eab8fa1ebdaecb15dffee878e" Oct 30 09:22:33.289522 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:22:33.289501 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94e093418121d5e79d5bee5710c4d5ee2f05ea5eab8fa1ebdaecb15dffee878e\": container with ID starting with 94e093418121d5e79d5bee5710c4d5ee2f05ea5eab8fa1ebdaecb15dffee878e not found: ID does not exist" containerID="94e093418121d5e79d5bee5710c4d5ee2f05ea5eab8fa1ebdaecb15dffee878e" Oct 30 09:22:33.289623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:33.289533 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e093418121d5e79d5bee5710c4d5ee2f05ea5eab8fa1ebdaecb15dffee878e"} err="failed to get container status \"94e093418121d5e79d5bee5710c4d5ee2f05ea5eab8fa1ebdaecb15dffee878e\": rpc error: code = NotFound desc = could not find container \"94e093418121d5e79d5bee5710c4d5ee2f05ea5eab8fa1ebdaecb15dffee878e\": container with ID starting with 94e093418121d5e79d5bee5710c4d5ee2f05ea5eab8fa1ebdaecb15dffee878e not found: ID does not exist" Oct 30 09:22:33.289623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:33.289552 5938 scope.go:122] "RemoveContainer" containerID="cf8c1adf1a5559733e7bfc0e89850c25d2d752e34f46cb256b78c54c5c8d8b23" Oct 30 09:22:33.289916 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:22:33.289899 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf8c1adf1a5559733e7bfc0e89850c25d2d752e34f46cb256b78c54c5c8d8b23\": container with ID starting with cf8c1adf1a5559733e7bfc0e89850c25d2d752e34f46cb256b78c54c5c8d8b23 not found: ID does not exist" containerID="cf8c1adf1a5559733e7bfc0e89850c25d2d752e34f46cb256b78c54c5c8d8b23" Oct 30 09:22:33.289985 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:33.289923 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf8c1adf1a5559733e7bfc0e89850c25d2d752e34f46cb256b78c54c5c8d8b23"} err="failed to get container status \"cf8c1adf1a5559733e7bfc0e89850c25d2d752e34f46cb256b78c54c5c8d8b23\": rpc error: code = NotFound desc = could not find container \"cf8c1adf1a5559733e7bfc0e89850c25d2d752e34f46cb256b78c54c5c8d8b23\": container with ID starting with cf8c1adf1a5559733e7bfc0e89850c25d2d752e34f46cb256b78c54c5c8d8b23 not found: ID does not exist" Oct 30 09:22:33.333596 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:33.333560 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz"] Oct 30 09:22:33.337485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:33.337448 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8d1106-openshift-default-599b8654cf-z9vlz"] Oct 30 09:22:34.260136 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:34.260098 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="523627cb-cbb0-4c8e-8783-5fe8203a57c3" path="/var/lib/kubelet/pods/523627cb-cbb0-4c8e-8783-5fe8203a57c3/volumes" Oct 30 09:22:35.605789 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:35.605751 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:22:35.606264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:35.605832 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:22:36.300617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:36.300560 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" event={"ID":"0daeb10f-9b40-4ecb-bccd-d03c06be57c7","Type":"ContainerStarted","Data":"27d71f7baa8f0db20c44110fc13599299b6aa2da2f4536bf654b2567c735abbc"} Oct 30 09:22:36.300976 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:36.300949 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:36.302923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:36.302894 5938 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-fw98g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:22:36.303088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:36.302948 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" podUID="0daeb10f-9b40-4ecb-bccd-d03c06be57c7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:22:36.368422 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:36.368363 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" podStartSLOduration=2.44484253 podStartE2EDuration="5.368342071s" totalImagesPullingTime="2.923499541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:36.357165023 +0000 UTC m=+1674.219870851" watchObservedRunningTime="2026-10-30 09:22:36.368342071 +0000 UTC m=+1674.231047898" Oct 30 09:22:37.311218 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:37.311187 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:22:45.812396 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:22:45.812355 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:22:51.470974 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.470923 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc"] Oct 30 09:22:51.471630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.471611 5938 cpu_manager.go:414] "containerMap: removing container" podUID="523627cb-cbb0-4c8e-8783-5fe8203a57c3" containerName="istio-proxy" Oct 30 09:22:51.471725 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.471634 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523627cb-cbb0-4c8e-8783-5fe8203a57c3" containerName="istio-proxy" Oct 30 09:22:51.471798 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.471788 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="523627cb-cbb0-4c8e-8783-5fe8203a57c3" containerName="istio-proxy" Oct 30 09:22:51.478852 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.478815 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.507956 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.507673 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8de308-openshift-default-dockercfg-9xrhg\"" Oct 30 09:22:51.510702 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.510184 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc"] Oct 30 09:22:51.536944 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.536907 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-credential-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.537153 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.536982 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-data\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.537153 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.537086 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs7w4\" (UniqueName: \"kubernetes.io/projected/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-kube-api-access-qs7w4\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.537153 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.537141 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-envoy\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.537626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.537170 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-token\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.537626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.537235 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-podinfo\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.537626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.537479 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-workload-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.537626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.537507 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-workload-certs\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.537626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.537554 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istiod-ca-cert\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.639322 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.639273 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-workload-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.639322 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.639314 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-workload-certs\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.639555 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.639351 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istiod-ca-cert\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.639555 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.639541 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-credential-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.639735 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.639615 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-data\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.639735 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.639713 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs7w4\" (UniqueName: \"kubernetes.io/projected/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-kube-api-access-qs7w4\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.639864 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.639772 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-envoy\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.639864 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.639801 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-token\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.639864 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.639845 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-workload-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.640032 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.639895 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-workload-certs\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.640032 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.639964 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-podinfo\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.640032 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.639974 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-credential-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.640139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.640053 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-data\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.640359 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.640341 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istiod-ca-cert\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.642574 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.642545 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-envoy\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.642806 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.642790 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-podinfo\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.661021 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.660988 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-token\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.661198 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.661158 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs7w4\" (UniqueName: \"kubernetes.io/projected/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-kube-api-access-qs7w4\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:51.815316 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:51.815214 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:52.156255 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:52.156227 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc"] Oct 30 09:22:52.158895 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:22:52.158865 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa1d623_e9d5_4c67_ad15_1fb90536fc46.slice/crio-ad8a25c87664ca0432ba832793ac29cce0fc62603a6c912cdc674d0b9f051d9a WatchSource:0}: Error finding container ad8a25c87664ca0432ba832793ac29cce0fc62603a6c912cdc674d0b9f051d9a: Status 404 returned error can't find the container with id ad8a25c87664ca0432ba832793ac29cce0fc62603a6c912cdc674d0b9f051d9a Oct 30 09:22:52.161923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:52.161881 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:22:52.162057 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:52.161964 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:22:52.162057 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:52.161997 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:22:52.417219 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:52.417104 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" event={"ID":"dfa1d623-e9d5-4c67-ad15-1fb90536fc46","Type":"ContainerStarted","Data":"72cfc67f3b9ff07cc11fcb33ce68645cd6352d0eceadc98b5c360f0eb173c219"} Oct 30 09:22:52.417219 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:52.417180 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" event={"ID":"dfa1d623-e9d5-4c67-ad15-1fb90536fc46","Type":"ContainerStarted","Data":"ad8a25c87664ca0432ba832793ac29cce0fc62603a6c912cdc674d0b9f051d9a"} Oct 30 09:22:52.467362 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:52.467292 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" podStartSLOduration=1.467266371 podStartE2EDuration="1.467266371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:52.462388831 +0000 UTC m=+1690.325094654" watchObservedRunningTime="2026-10-30 09:22:52.467266371 +0000 UTC m=+1690.329972193" Oct 30 09:22:52.816140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:52.816037 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:53.818070 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:53.818038 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 09:22:53.818511 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:53.818092 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" podUID="dfa1d623-e9d5-4c67-ad15-1fb90536fc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 09:22:54.816368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:54.816325 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 09:22:54.816664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:54.816411 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" podUID="dfa1d623-e9d5-4c67-ad15-1fb90536fc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 09:22:55.815683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:55.815626 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 09:22:55.816129 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:55.815703 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" podUID="dfa1d623-e9d5-4c67-ad15-1fb90536fc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 09:22:56.816599 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:56.816548 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 09:22:56.817072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:56.816660 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" podUID="dfa1d623-e9d5-4c67-ad15-1fb90536fc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 09:22:57.815874 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:57.815833 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 09:22:57.816096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:57.815900 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" podUID="dfa1d623-e9d5-4c67-ad15-1fb90536fc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 09:22:58.821899 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:58.821865 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:58.822357 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:58.822332 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:22:58.823248 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:22:58.823228 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:23:15.812211 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:23:15.812171 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:23:16.596959 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:16.596909 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56"] Oct 30 09:23:31.620384 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.620302 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" podUID="e91e4e53-b0ac-491a-8baf-4550976139aa" containerName="main" containerID="cri-o://545b953d9ce56227768585815cc99a8ff75b43892e223fa711792356a712305e" gracePeriod=45 Oct 30 09:23:31.864033 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.863997 5938 generic.go:356] "Generic (PLEG): container finished" podID="e91e4e53-b0ac-491a-8baf-4550976139aa" containerID="545b953d9ce56227768585815cc99a8ff75b43892e223fa711792356a712305e" exitCode=0 Oct 30 09:23:31.864234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.864157 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" event={"ID":"e91e4e53-b0ac-491a-8baf-4550976139aa","Type":"ContainerDied","Data":"545b953d9ce56227768585815cc99a8ff75b43892e223fa711792356a712305e"} Oct 30 09:23:31.914419 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.914374 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:23:31.962805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.962772 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-home\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-home\") pod \"e91e4e53-b0ac-491a-8baf-4550976139aa\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " Oct 30 09:23:31.963004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.962907 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-dshm\") pod \"e91e4e53-b0ac-491a-8baf-4550976139aa\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " Oct 30 09:23:31.963004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.962987 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-model-cache\") pod \"e91e4e53-b0ac-491a-8baf-4550976139aa\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " Oct 30 09:23:31.963138 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.963019 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e91e4e53-b0ac-491a-8baf-4550976139aa-kube-api-access-gz8ds\" (UniqueName: \"kubernetes.io/projected/e91e4e53-b0ac-491a-8baf-4550976139aa-kube-api-access-gz8ds\") pod \"e91e4e53-b0ac-491a-8baf-4550976139aa\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " Oct 30 09:23:31.963138 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.963109 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-home" pod "e91e4e53-b0ac-491a-8baf-4550976139aa" (UID: "e91e4e53-b0ac-491a-8baf-4550976139aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:23:31.963244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.963144 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-tmp-dir\") pod \"e91e4e53-b0ac-491a-8baf-4550976139aa\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " Oct 30 09:23:31.963244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.963186 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e91e4e53-b0ac-491a-8baf-4550976139aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/e91e4e53-b0ac-491a-8baf-4550976139aa-tls-certs\") pod \"e91e4e53-b0ac-491a-8baf-4550976139aa\" (UID: \"e91e4e53-b0ac-491a-8baf-4550976139aa\") " Oct 30 09:23:31.963350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.963278 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-model-cache" pod "e91e4e53-b0ac-491a-8baf-4550976139aa" (UID: "e91e4e53-b0ac-491a-8baf-4550976139aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:23:31.963897 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.963468 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-tmp-dir" pod "e91e4e53-b0ac-491a-8baf-4550976139aa" (UID: "e91e4e53-b0ac-491a-8baf-4550976139aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:23:31.964137 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.964114 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:23:31.964214 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.964143 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:23:31.964214 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.964158 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:23:31.966383 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.966347 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e91e4e53-b0ac-491a-8baf-4550976139aa-kube-api-access-gz8ds" pod "e91e4e53-b0ac-491a-8baf-4550976139aa" (UID: "e91e4e53-b0ac-491a-8baf-4550976139aa"). InnerVolumeSpecName "kube-api-access-gz8ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:23:31.966749 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.966698 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-dshm" pod "e91e4e53-b0ac-491a-8baf-4550976139aa" (UID: "e91e4e53-b0ac-491a-8baf-4550976139aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:23:31.966749 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:31.966723 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e91e4e53-b0ac-491a-8baf-4550976139aa-tls-certs" pod "e91e4e53-b0ac-491a-8baf-4550976139aa" (UID: "e91e4e53-b0ac-491a-8baf-4550976139aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:23:32.065273 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.065229 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91e4e53-b0ac-491a-8baf-4550976139aa-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:23:32.065273 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.065264 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz8ds\" (UniqueName: \"kubernetes.io/projected/e91e4e53-b0ac-491a-8baf-4550976139aa-kube-api-access-gz8ds\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:23:32.065273 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.065281 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91e4e53-b0ac-491a-8baf-4550976139aa-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:23:32.124070 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.123790 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz"] Oct 30 09:23:32.124574 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.124555 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e91e4e53-b0ac-491a-8baf-4550976139aa" containerName="main" Oct 30 09:23:32.124683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.124577 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e91e4e53-b0ac-491a-8baf-4550976139aa" containerName="main" Oct 30 09:23:32.124683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.124608 5938 cpu_manager.go:414] "containerMap: removing container" podUID="523627cb-cbb0-4c8e-8783-5fe8203a57c3" containerName="istio-proxy" Oct 30 09:23:32.124683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.124615 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523627cb-cbb0-4c8e-8783-5fe8203a57c3" containerName="istio-proxy" Oct 30 09:23:32.124795 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.124756 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e91e4e53-b0ac-491a-8baf-4550976139aa" containerName="main" Oct 30 09:23:32.124795 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.124765 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="523627cb-cbb0-4c8e-8783-5fe8203a57c3" containerName="istio-proxy" Oct 30 09:23:32.132666 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.128791 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.153971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.153436 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9371b7-openshift-default-dockercfg-fccbl\"" Oct 30 09:23:32.158071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.158034 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz"] Oct 30 09:23:32.166808 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.166750 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf8q4\" (UniqueName: \"kubernetes.io/projected/58a20fda-9d11-47ca-887d-e1dac99e53a2-kube-api-access-wf8q4\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.167012 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.166825 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-workload-socket\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.167012 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.166960 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-workload-certs\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.167012 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.167007 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-token\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.167189 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.167149 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-data\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.167243 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.167201 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-envoy\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.167296 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.167268 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-credential-socket\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.167363 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.167340 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-podinfo\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.167435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.167394 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a20fda-9d11-47ca-887d-e1dac99e53a2-istiod-ca-cert\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.339388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.331141 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-workload-certs\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.339388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.331219 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-token\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.339388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.331301 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-data\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.339388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.331353 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-envoy\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.339388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.331401 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-credential-socket\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.339388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.331484 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-podinfo\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.339388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.331526 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a20fda-9d11-47ca-887d-e1dac99e53a2-istiod-ca-cert\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.339388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.331700 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf8q4\" (UniqueName: \"kubernetes.io/projected/58a20fda-9d11-47ca-887d-e1dac99e53a2-kube-api-access-wf8q4\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.339388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.331748 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-workload-socket\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.339388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.332293 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-workload-socket\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.339388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.332680 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-workload-certs\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.339388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.333206 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-data\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.339388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.338132 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-credential-socket\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.339388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.338997 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a20fda-9d11-47ca-887d-e1dac99e53a2-istiod-ca-cert\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.354117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.354043 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-envoy\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.359031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.358990 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-podinfo\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.369694 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.366352 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf8q4\" (UniqueName: \"kubernetes.io/projected/58a20fda-9d11-47ca-887d-e1dac99e53a2-kube-api-access-wf8q4\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.373747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.373660 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-token\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.461383 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.460859 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:32.809681 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.809207 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz"] Oct 30 09:23:32.821089 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.821037 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:23:32.821261 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.821196 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:23:32.821261 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.821254 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:23:32.853504 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.853447 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" podUID="e91e4e53-b0ac-491a-8baf-4550976139aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:23:32.873744 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.873708 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" event={"ID":"e91e4e53-b0ac-491a-8baf-4550976139aa","Type":"ContainerDied","Data":"aa433a9ea1088c19377f4b2e543e411753b081cfc800087bfe3cce03623db4b6"} Oct 30 09:23:32.873928 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.873750 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56" Oct 30 09:23:32.873928 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.873777 5938 scope.go:122] "RemoveContainer" containerID="545b953d9ce56227768585815cc99a8ff75b43892e223fa711792356a712305e" Oct 30 09:23:32.875983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.875948 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" event={"ID":"58a20fda-9d11-47ca-887d-e1dac99e53a2","Type":"ContainerStarted","Data":"b266c118ff3fb25eb576a75280b3986480de809a030941127513be697a7f723b"} Oct 30 09:23:32.946739 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.946657 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56"] Oct 30 09:23:32.968919 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:32.968871 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8eada0cf/e2e-lifecycle-model-8eada0cf-kserve-85bd4fdf9c-8tm56"] Oct 30 09:23:33.885964 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:33.885925 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" event={"ID":"58a20fda-9d11-47ca-887d-e1dac99e53a2","Type":"ContainerStarted","Data":"1e7efcb9aded3ee94affc28ff686a36b3e75de2e348dce3a8d2c5ff5be0916fe"} Oct 30 09:23:34.260380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:34.260286 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e91e4e53-b0ac-491a-8baf-4550976139aa" path="/var/lib/kubelet/pods/e91e4e53-b0ac-491a-8baf-4550976139aa/volumes" Oct 30 09:23:34.461684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:34.461634 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:34.468005 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:34.467971 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:34.513309 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:34.513205 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" podStartSLOduration=2.5131892750000002 podStartE2EDuration="2.513189275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:33.959223428 +0000 UTC m=+1731.821929249" watchObservedRunningTime="2026-10-30 09:23:34.513189275 +0000 UTC m=+1732.375895096" Oct 30 09:23:34.895018 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:34.894979 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:34.896317 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:34.896299 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:23:39.082281 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:39.082246 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt"] Oct 30 09:23:39.087891 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:39.087856 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" Oct 30 09:23:39.105173 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:39.105137 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt"] Oct 30 09:23:39.115624 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:39.115576 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9371b7\"" Oct 30 09:23:39.115821 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:39.115704 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9371b7-dockercfg-smg8n\"" Oct 30 09:23:39.230356 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:39.230054 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb08123-121f-4432-93fb-c4b7a5f8bb6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt\" (UID: \"abb08123-121f-4432-93fb-c4b7a5f8bb6a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" Oct 30 09:23:39.230356 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:39.230152 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm25n\" (UniqueName: \"kubernetes.io/projected/abb08123-121f-4432-93fb-c4b7a5f8bb6a-kube-api-access-fm25n\") pod \"payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt\" (UID: \"abb08123-121f-4432-93fb-c4b7a5f8bb6a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" Oct 30 09:23:39.333273 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:39.333165 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb08123-121f-4432-93fb-c4b7a5f8bb6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt\" (UID: \"abb08123-121f-4432-93fb-c4b7a5f8bb6a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" Oct 30 09:23:39.333571 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:39.333547 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm25n\" (UniqueName: \"kubernetes.io/projected/abb08123-121f-4432-93fb-c4b7a5f8bb6a-kube-api-access-fm25n\") pod \"payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt\" (UID: \"abb08123-121f-4432-93fb-c4b7a5f8bb6a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" Oct 30 09:23:39.335293 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:39.335261 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb08123-121f-4432-93fb-c4b7a5f8bb6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt\" (UID: \"abb08123-121f-4432-93fb-c4b7a5f8bb6a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" Oct 30 09:23:39.355929 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:39.355895 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm25n\" (UniqueName: \"kubernetes.io/projected/abb08123-121f-4432-93fb-c4b7a5f8bb6a-kube-api-access-fm25n\") pod \"payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt\" (UID: \"abb08123-121f-4432-93fb-c4b7a5f8bb6a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" Oct 30 09:23:39.405567 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:39.405512 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" Oct 30 09:23:39.728669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:39.728642 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt"] Oct 30 09:23:39.730093 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:23:39.730055 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb08123_121f_4432_93fb_c4b7a5f8bb6a.slice/crio-8dbe6d1b31db75367ddb914247564bf9dd2ac7469f2d384cfdefca127d547a08 WatchSource:0}: Error finding container 8dbe6d1b31db75367ddb914247564bf9dd2ac7469f2d384cfdefca127d547a08: Status 404 returned error can't find the container with id 8dbe6d1b31db75367ddb914247564bf9dd2ac7469f2d384cfdefca127d547a08 Oct 30 09:23:39.939012 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:39.938963 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" event={"ID":"abb08123-121f-4432-93fb-c4b7a5f8bb6a","Type":"ContainerStarted","Data":"8dbe6d1b31db75367ddb914247564bf9dd2ac7469f2d384cfdefca127d547a08"} Oct 30 09:23:41.562757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.562725 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc"] Oct 30 09:23:41.575276 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.575250 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:23:41.587408 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.587381 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc"] Oct 30 09:23:41.596736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.596704 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-cbe974-serving-cert\"" Oct 30 09:23:41.671333 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.671285 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc\" (UID: \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:23:41.671333 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.671340 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc\" (UID: \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:23:41.671862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.671518 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npmwq\" (UniqueName: \"kubernetes.io/projected/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-kube-api-access-npmwq\") pod \"maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc\" (UID: \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:23:41.671862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.671739 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-metrics-certs\") pod \"maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc\" (UID: \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:23:41.773999 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.773955 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npmwq\" (UniqueName: \"kubernetes.io/projected/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-kube-api-access-npmwq\") pod \"maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc\" (UID: \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:23:41.774210 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.774106 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-metrics-certs\") pod \"maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc\" (UID: \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:23:41.776334 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.774373 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc\" (UID: \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:23:41.776334 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.774440 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc\" (UID: \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:23:41.776334 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.775683 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc\" (UID: \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:23:41.780256 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.780226 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-metrics-certs\") pod \"maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc\" (UID: \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:23:41.782433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.782365 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc\" (UID: \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:23:41.797873 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.797843 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npmwq\" (UniqueName: \"kubernetes.io/projected/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-kube-api-access-npmwq\") pod \"maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc\" (UID: \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:23:41.895419 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:41.895388 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:23:42.210446 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:42.210414 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc"] Oct 30 09:23:42.211806 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:23:42.211772 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946ef50b_d7e0_4b57_a1f3_d91120bee9e5.slice/crio-ea7def1be610f80348a0f449a92607a5b5034e83a29a37d5bc0515ece95c99ed WatchSource:0}: Error finding container ea7def1be610f80348a0f449a92607a5b5034e83a29a37d5bc0515ece95c99ed: Status 404 returned error can't find the container with id ea7def1be610f80348a0f449a92607a5b5034e83a29a37d5bc0515ece95c99ed Oct 30 09:23:42.964692 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:42.964648 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" event={"ID":"946ef50b-d7e0-4b57-a1f3-d91120bee9e5","Type":"ContainerStarted","Data":"86ebb130e50e829a1f210e34629e90b707a90036f68f2ca2495dbc4e84ece38b"} Oct 30 09:23:42.965198 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:42.964717 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" event={"ID":"946ef50b-d7e0-4b57-a1f3-d91120bee9e5","Type":"ContainerStarted","Data":"ea7def1be610f80348a0f449a92607a5b5034e83a29a37d5bc0515ece95c99ed"} Oct 30 09:23:42.965198 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:42.964744 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:23:42.966385 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:42.966359 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" event={"ID":"abb08123-121f-4432-93fb-c4b7a5f8bb6a","Type":"ContainerStarted","Data":"eb0108bbc44706e727090725bf3acad3c344ca011e4b85977f52ea6e4b795ae1"} Oct 30 09:23:42.966507 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:42.966483 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" Oct 30 09:23:43.013150 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:43.013097 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" podStartSLOduration=1.665285893 podStartE2EDuration="2.013083297s" totalImagesPullingTime="347.797404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:43.011547569 +0000 UTC m=+1740.874253391" watchObservedRunningTime="2026-10-30 09:23:43.013083297 +0000 UTC m=+1740.875789117" Oct 30 09:23:43.062712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:43.062629 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" podStartSLOduration=1.936097749 podStartE2EDuration="4.062609362s" totalImagesPullingTime="2.126511613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:43.054124135 +0000 UTC m=+1740.916829957" watchObservedRunningTime="2026-10-30 09:23:43.062609362 +0000 UTC m=+1740.925315183" Oct 30 09:23:45.815517 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:23:45.815482 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:23:48.980069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:48.980033 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:23:53.977002 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:53.976972 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" Oct 30 09:23:56.651863 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.651816 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw"] Oct 30 09:23:56.659452 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.659422 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:23:56.676403 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.676367 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw"] Oct 30 09:23:56.688633 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.686089 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-962e2e-serving-cert\"" Oct 30 09:23:56.718675 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.718636 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m44tw\" (UniqueName: \"kubernetes.io/projected/e64333ed-ee2e-4099-82ca-6ae5f140058f-kube-api-access-m44tw\") pod \"maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw\" (UID: \"e64333ed-ee2e-4099-82ca-6ae5f140058f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:23:56.718896 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.718737 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64333ed-ee2e-4099-82ca-6ae5f140058f-metrics-certs\") pod \"maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw\" (UID: \"e64333ed-ee2e-4099-82ca-6ae5f140058f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:23:56.718896 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.718774 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e64333ed-ee2e-4099-82ca-6ae5f140058f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw\" (UID: \"e64333ed-ee2e-4099-82ca-6ae5f140058f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:23:56.719032 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.718905 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64333ed-ee2e-4099-82ca-6ae5f140058f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw\" (UID: \"e64333ed-ee2e-4099-82ca-6ae5f140058f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:23:56.820797 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.820761 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m44tw\" (UniqueName: \"kubernetes.io/projected/e64333ed-ee2e-4099-82ca-6ae5f140058f-kube-api-access-m44tw\") pod \"maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw\" (UID: \"e64333ed-ee2e-4099-82ca-6ae5f140058f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:23:56.821001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.820829 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64333ed-ee2e-4099-82ca-6ae5f140058f-metrics-certs\") pod \"maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw\" (UID: \"e64333ed-ee2e-4099-82ca-6ae5f140058f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:23:56.821001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.820853 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e64333ed-ee2e-4099-82ca-6ae5f140058f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw\" (UID: \"e64333ed-ee2e-4099-82ca-6ae5f140058f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:23:56.821001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.820950 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64333ed-ee2e-4099-82ca-6ae5f140058f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw\" (UID: \"e64333ed-ee2e-4099-82ca-6ae5f140058f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:23:56.821653 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.821629 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64333ed-ee2e-4099-82ca-6ae5f140058f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw\" (UID: \"e64333ed-ee2e-4099-82ca-6ae5f140058f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:23:56.823675 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.823654 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64333ed-ee2e-4099-82ca-6ae5f140058f-metrics-certs\") pod \"maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw\" (UID: \"e64333ed-ee2e-4099-82ca-6ae5f140058f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:23:56.823834 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.823802 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e64333ed-ee2e-4099-82ca-6ae5f140058f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw\" (UID: \"e64333ed-ee2e-4099-82ca-6ae5f140058f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:23:56.841558 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.841528 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m44tw\" (UniqueName: \"kubernetes.io/projected/e64333ed-ee2e-4099-82ca-6ae5f140058f-kube-api-access-m44tw\") pod \"maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw\" (UID: \"e64333ed-ee2e-4099-82ca-6ae5f140058f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:23:56.980507 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:56.980349 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:23:57.374865 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:57.374837 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw"] Oct 30 09:23:57.377518 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:23:57.377478 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64333ed_ee2e_4099_82ca_6ae5f140058f.slice/crio-81fe6d4f8474cc1a5db300c919c2adff790ae64e6aa50abcb18e76d40263bc8c WatchSource:0}: Error finding container 81fe6d4f8474cc1a5db300c919c2adff790ae64e6aa50abcb18e76d40263bc8c: Status 404 returned error can't find the container with id 81fe6d4f8474cc1a5db300c919c2adff790ae64e6aa50abcb18e76d40263bc8c Oct 30 09:23:58.112353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:58.112183 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" event={"ID":"e64333ed-ee2e-4099-82ca-6ae5f140058f","Type":"ContainerStarted","Data":"5ac1c42b48ffe4c0352119e18b174b0f6aaa942d16c024e899848b1a18382d5a"} Oct 30 09:23:58.112353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:58.112253 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" event={"ID":"e64333ed-ee2e-4099-82ca-6ae5f140058f","Type":"ContainerStarted","Data":"81fe6d4f8474cc1a5db300c919c2adff790ae64e6aa50abcb18e76d40263bc8c"} Oct 30 09:23:58.112353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:58.112296 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:23:58.156241 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:23:58.156154 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" podStartSLOduration=1.8403775310000001 podStartE2EDuration="2.156130559s" totalImagesPullingTime="315.753028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:58.15393756 +0000 UTC m=+1756.016643385" watchObservedRunningTime="2026-10-30 09:23:58.156130559 +0000 UTC m=+1756.018836384" Oct 30 09:24:04.126444 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:04.126415 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:24:07.970369 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:07.970331 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x"] Oct 30 09:24:07.974994 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:07.974972 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" Oct 30 09:24:07.992627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:07.992573 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x"] Oct 30 09:24:08.002011 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:08.001976 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4fc2e2\"" Oct 30 09:24:08.039701 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:08.031948 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4fc2e2-dockercfg-zg9w9\"" Oct 30 09:24:08.039701 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:08.034270 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg"] Oct 30 09:24:08.039701 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:08.039270 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb41e22d-x25rg"] Oct 30 09:24:08.100615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:08.085164 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jws5n\" (UniqueName: \"kubernetes.io/projected/0eea09cb-d491-4c49-8013-000393c05e5f-kube-api-access-jws5n\") pod \"payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x\" (UID: \"0eea09cb-d491-4c49-8013-000393c05e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" Oct 30 09:24:08.210128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:08.209046 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eea09cb-d491-4c49-8013-000393c05e5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x\" (UID: \"0eea09cb-d491-4c49-8013-000393c05e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" Oct 30 09:24:08.269669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:08.267524 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="515e16d8-c8ca-4e60-b1ae-462c41c614ab" path="/var/lib/kubelet/pods/515e16d8-c8ca-4e60-b1ae-462c41c614ab/volumes" Oct 30 09:24:08.311077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:08.311039 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eea09cb-d491-4c49-8013-000393c05e5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x\" (UID: \"0eea09cb-d491-4c49-8013-000393c05e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" Oct 30 09:24:08.311295 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:08.311154 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jws5n\" (UniqueName: \"kubernetes.io/projected/0eea09cb-d491-4c49-8013-000393c05e5f-kube-api-access-jws5n\") pod \"payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x\" (UID: \"0eea09cb-d491-4c49-8013-000393c05e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" Oct 30 09:24:08.311805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:08.311785 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eea09cb-d491-4c49-8013-000393c05e5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x\" (UID: \"0eea09cb-d491-4c49-8013-000393c05e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" Oct 30 09:24:08.346940 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:08.346912 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jws5n\" (UniqueName: \"kubernetes.io/projected/0eea09cb-d491-4c49-8013-000393c05e5f-kube-api-access-jws5n\") pod \"payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x\" (UID: \"0eea09cb-d491-4c49-8013-000393c05e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" Oct 30 09:24:08.592782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:08.592675 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" Oct 30 09:24:08.902862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:08.902826 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x"] Oct 30 09:24:08.904410 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:24:08.904384 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eea09cb_d491_4c49_8013_000393c05e5f.slice/crio-38d2499d04bd87ed935a6e4ea4bc8f99d5460f027e29c20da2b52cadc905a177 WatchSource:0}: Error finding container 38d2499d04bd87ed935a6e4ea4bc8f99d5460f027e29c20da2b52cadc905a177: Status 404 returned error can't find the container with id 38d2499d04bd87ed935a6e4ea4bc8f99d5460f027e29c20da2b52cadc905a177 Oct 30 09:24:09.227983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:09.227866 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" event={"ID":"0eea09cb-d491-4c49-8013-000393c05e5f","Type":"ContainerStarted","Data":"7dcf78b820fba9623ddda4004416f1264385e6d80182139d9b169585cc375a4c"} Oct 30 09:24:09.227983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:09.227930 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" event={"ID":"0eea09cb-d491-4c49-8013-000393c05e5f","Type":"ContainerStarted","Data":"38d2499d04bd87ed935a6e4ea4bc8f99d5460f027e29c20da2b52cadc905a177"} Oct 30 09:24:09.227983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:09.227960 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" Oct 30 09:24:09.269992 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:09.269934 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" podStartSLOduration=2.2699186239999998 podStartE2EDuration="2.269918624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:24:09.267918013 +0000 UTC m=+1767.130623835" watchObservedRunningTime="2026-10-30 09:24:09.269918624 +0000 UTC m=+1767.132624511" Oct 30 09:24:12.772268 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:12.772222 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2"] Oct 30 09:24:12.777215 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:12.777187 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:24:12.793520 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:12.793436 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-356553-serving-cert\"" Oct 30 09:24:12.810121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:12.810089 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2"] Oct 30 09:24:12.884211 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:12.884170 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2\" (UID: \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:24:12.884418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:12.884343 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-metrics-certs\") pod \"maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2\" (UID: \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:24:12.884418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:12.884383 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs776\" (UniqueName: \"kubernetes.io/projected/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-kube-api-access-cs776\") pod \"maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2\" (UID: \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:24:12.884575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:12.884555 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2\" (UID: \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:24:12.986498 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:12.986462 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2\" (UID: \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:24:12.986699 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:12.986607 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2\" (UID: \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:24:12.986852 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:12.986832 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-metrics-certs\") pod \"maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2\" (UID: \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:24:12.986908 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:12.986891 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs776\" (UniqueName: \"kubernetes.io/projected/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-kube-api-access-cs776\") pod \"maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2\" (UID: \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:24:12.987406 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:12.987336 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2\" (UID: \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:24:12.989478 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:12.989451 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-metrics-certs\") pod \"maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2\" (UID: \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:24:12.989478 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:12.989467 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2\" (UID: \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:24:13.009149 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:13.009112 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs776\" (UniqueName: \"kubernetes.io/projected/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-kube-api-access-cs776\") pod \"maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2\" (UID: \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:24:13.091640 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:13.091477 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:24:13.401913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:13.401877 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2"] Oct 30 09:24:13.403358 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:24:13.403331 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb18f58e_ec28_4e52_a9c0_f8bee67cf18a.slice/crio-bc614316b32f292c39d1e33bf2728baaaf76cb26bb78ba0124061c3674345e13 WatchSource:0}: Error finding container bc614316b32f292c39d1e33bf2728baaaf76cb26bb78ba0124061c3674345e13: Status 404 returned error can't find the container with id bc614316b32f292c39d1e33bf2728baaaf76cb26bb78ba0124061c3674345e13 Oct 30 09:24:14.280047 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:14.279975 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" event={"ID":"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a","Type":"ContainerStarted","Data":"e0ca2787b2f6d867f97d65a677af3a77624f389749f99dc39220e6c6ea16e14a"} Oct 30 09:24:14.280660 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:14.280076 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" event={"ID":"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a","Type":"ContainerStarted","Data":"bc614316b32f292c39d1e33bf2728baaaf76cb26bb78ba0124061c3674345e13"} Oct 30 09:24:14.282777 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:14.282753 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:24:14.334203 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:14.334129 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" podStartSLOduration=1.99715069 podStartE2EDuration="2.334103154s" totalImagesPullingTime="336.952464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:24:14.327189069 +0000 UTC m=+1772.189894894" watchObservedRunningTime="2026-10-30 09:24:14.334103154 +0000 UTC m=+1772.196808981" Oct 30 09:24:15.810735 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:24:15.810697 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:24:20.238568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:20.238536 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" Oct 30 09:24:20.294848 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:20.294805 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:24:32.449017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:32.448920 5938 generic.go:356] "Generic (PLEG): container finished" podID="58a20fda-9d11-47ca-887d-e1dac99e53a2" containerID="1e7efcb9aded3ee94affc28ff686a36b3e75de2e348dce3a8d2c5ff5be0916fe" exitCode=0 Oct 30 09:24:32.449553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:32.449045 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" event={"ID":"58a20fda-9d11-47ca-887d-e1dac99e53a2","Type":"ContainerDied","Data":"1e7efcb9aded3ee94affc28ff686a36b3e75de2e348dce3a8d2c5ff5be0916fe"} Oct 30 09:24:32.450158 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:32.449631 5938 scope.go:122] "RemoveContainer" containerID="1e7efcb9aded3ee94affc28ff686a36b3e75de2e348dce3a8d2c5ff5be0916fe" Oct 30 09:24:32.453490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:32.452126 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:24:32.453490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:32.452241 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:24:32.453490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:32.452292 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:24:33.469822 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:33.469774 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" event={"ID":"58a20fda-9d11-47ca-887d-e1dac99e53a2","Type":"ContainerStarted","Data":"7e437cafd311955f2b446862532a3472307d24d1bc2d5f23b4b04ce09c0debc0"} Oct 30 09:24:34.461799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:34.461764 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:24:34.462895 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:34.462869 5938 patch_prober.go:29] interesting pod/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 09:24:34.463025 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:34.462925 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" podUID="58a20fda-9d11-47ca-887d-e1dac99e53a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 09:24:34.895561 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:34.895517 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:24:35.496615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:35.479761 5938 patch_prober.go:29] interesting pod/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 09:24:35.496615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:35.479851 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" podUID="58a20fda-9d11-47ca-887d-e1dac99e53a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 09:24:36.462244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:36.462206 5938 patch_prober.go:29] interesting pod/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 09:24:36.462847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:36.462289 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" podUID="58a20fda-9d11-47ca-887d-e1dac99e53a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 09:24:37.461896 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:37.461856 5938 patch_prober.go:29] interesting pod/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 09:24:37.462131 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:37.461962 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" podUID="58a20fda-9d11-47ca-887d-e1dac99e53a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 09:24:38.462010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:38.461972 5938 patch_prober.go:29] interesting pod/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 09:24:38.462435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:38.462042 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" podUID="58a20fda-9d11-47ca-887d-e1dac99e53a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 09:24:39.467519 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:39.467486 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:24:39.468835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:24:39.468810 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:24:45.811943 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:24:45.811897 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:25:15.813624 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:25:15.813517 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:25:20.462303 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:20.462268 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9"] Oct 30 09:25:20.462798 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:20.462505 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" podUID="7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" containerName="istio-proxy" containerID="cri-o://a93a339ea2ca46429c22c24bbe35df2970b5eda6c52d6efcf8e1988ddb1aa504" gracePeriod=30 Oct 30 09:25:25.761294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.761255 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:25:25.836791 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.836749 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-workload-socket\") pod \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " Oct 30 09:25:25.837006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.836845 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-kube-api-access-7hh8j\" (UniqueName: \"kubernetes.io/projected/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-kube-api-access-7hh8j\") pod \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " Oct 30 09:25:25.837006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.836904 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-envoy\") pod \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " Oct 30 09:25:25.837145 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.837033 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-data\") pod \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " Oct 30 09:25:25.837145 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.837077 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-token\" (UniqueName: \"kubernetes.io/projected/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-token\") pod \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " Oct 30 09:25:25.837298 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.837275 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-workload-socket" pod "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" (UID: "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:25.837411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.837286 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-credential-socket\") pod \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " Oct 30 09:25:25.837411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.837385 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istiod-ca-cert\") pod \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " Oct 30 09:25:25.837527 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.837440 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-workload-certs\") pod \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " Oct 30 09:25:25.837581 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.837531 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-podinfo\") pod \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\" (UID: \"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1\") " Oct 30 09:25:25.837581 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.837555 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-credential-socket" pod "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" (UID: "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:25.838231 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.838196 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-workload-certs" pod "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" (UID: "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:25.838401 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.838371 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istiod-ca-cert" pod "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" (UID: "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:25.838680 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.838647 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-data" pod "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" (UID: "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:25.839622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.839580 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:25.839769 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.839754 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:25.839890 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.839875 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:25.840049 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.840036 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:25.840139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.840127 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:25.841372 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.841346 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-podinfo" pod "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" (UID: "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:25.843227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.843068 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-kube-api-access-7hh8j" pod "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" (UID: "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1"). InnerVolumeSpecName "kube-api-access-7hh8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:25.843336 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.843230 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-token" pod "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" (UID: "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:25.845082 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.845054 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-envoy" pod "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" (UID: "7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:25.942469 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.942405 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:25.942810 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.942788 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hh8j\" (UniqueName: \"kubernetes.io/projected/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-kube-api-access-7hh8j\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:25.942946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.942929 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:25.943055 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:25.943042 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:26.078442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.078349 5938 generic.go:356] "Generic (PLEG): container finished" podID="7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" containerID="a93a339ea2ca46429c22c24bbe35df2970b5eda6c52d6efcf8e1988ddb1aa504" exitCode=0 Oct 30 09:25:26.078880 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.078727 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" Oct 30 09:25:26.080007 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.078749 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" event={"ID":"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1","Type":"ContainerDied","Data":"a93a339ea2ca46429c22c24bbe35df2970b5eda6c52d6efcf8e1988ddb1aa504"} Oct 30 09:25:26.080007 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.079822 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9" event={"ID":"7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1","Type":"ContainerDied","Data":"8d9b4306e479b7231be906f9801aaa50e38faf88cd867ab5f09aca83e4f2df46"} Oct 30 09:25:26.080007 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.079863 5938 scope.go:122] "RemoveContainer" containerID="a93a339ea2ca46429c22c24bbe35df2970b5eda6c52d6efcf8e1988ddb1aa504" Oct 30 09:25:26.105612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.100669 5938 scope.go:122] "RemoveContainer" containerID="a93a339ea2ca46429c22c24bbe35df2970b5eda6c52d6efcf8e1988ddb1aa504" Oct 30 09:25:26.105612 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:25:26.102283 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a93a339ea2ca46429c22c24bbe35df2970b5eda6c52d6efcf8e1988ddb1aa504\": container with ID starting with a93a339ea2ca46429c22c24bbe35df2970b5eda6c52d6efcf8e1988ddb1aa504 not found: ID does not exist" containerID="a93a339ea2ca46429c22c24bbe35df2970b5eda6c52d6efcf8e1988ddb1aa504" Oct 30 09:25:26.105612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.102323 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93a339ea2ca46429c22c24bbe35df2970b5eda6c52d6efcf8e1988ddb1aa504"} err="failed to get container status \"a93a339ea2ca46429c22c24bbe35df2970b5eda6c52d6efcf8e1988ddb1aa504\": rpc error: code = NotFound desc = could not find container \"a93a339ea2ca46429c22c24bbe35df2970b5eda6c52d6efcf8e1988ddb1aa504\": container with ID starting with a93a339ea2ca46429c22c24bbe35df2970b5eda6c52d6efcf8e1988ddb1aa504 not found: ID does not exist" Oct 30 09:25:26.161026 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.160985 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9"] Oct 30 09:25:26.164258 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.164220 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-qzwf9"] Oct 30 09:25:26.216994 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.216956 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4"] Oct 30 09:25:26.217244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.217214 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" podUID="4fce2865-c606-45f4-84d0-6691b9768fc7" containerName="istio-proxy" containerID="cri-o://e8cdf4906214880aaabfc38e9a3691a0ac56a57cf4e55858f373b39b6af6f15f" gracePeriod=30 Oct 30 09:25:26.231711 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.231670 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547"] Oct 30 09:25:26.231997 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.231968 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerName="istio-proxy" containerID="cri-o://f0569446e9f952d35b24f776b142855f8169a78697abd78b1f64a2e4737c8ed3" gracePeriod=30 Oct 30 09:25:26.234757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.234564 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc"] Oct 30 09:25:26.243624 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.235660 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" podUID="929a1bc8-672b-49b2-8344-9253178bd699" containerName="istio-proxy" containerID="cri-o://0e9c7d5f552a1be49b2ea07326a99317b554c810cc2654b3e766994d73e4267d" gracePeriod=30 Oct 30 09:25:26.243624 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.238423 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh"] Oct 30 09:25:26.243624 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.238668 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" podUID="cda8e58c-2364-4edb-be89-36fc16609681" containerName="istio-proxy" containerID="cri-o://243f303b17815d116e7f0eac32302a4befd813562dd9217ca1def3fe16ebb129" gracePeriod=30 Oct 30 09:25:26.268879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:26.268714 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" path="/var/lib/kubelet/pods/7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1/volumes" Oct 30 09:25:31.572285 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.572247 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:25:31.587143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.587046 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:25:31.654743 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.654706 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:25:31.662465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.662432 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:25:31.743796 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.743759 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda8e58c-2364-4edb-be89-36fc16609681-kube-api-access-fpzwd\" (UniqueName: \"kubernetes.io/projected/cda8e58c-2364-4edb-be89-36fc16609681-kube-api-access-fpzwd\") pod \"cda8e58c-2364-4edb-be89-36fc16609681\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " Oct 30 09:25:31.744085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.743850 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-data\") pod \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " Oct 30 09:25:31.744085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.743883 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-credential-socket\") pod \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " Oct 30 09:25:31.744085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.743925 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-podinfo\") pod \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " Oct 30 09:25:31.744085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.743959 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-kube-api-access-6zrc8\" (UniqueName: \"kubernetes.io/projected/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-kube-api-access-6zrc8\") pod \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " Oct 30 09:25:31.746000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.744447 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-podinfo\") pod \"4fce2865-c606-45f4-84d0-6691b9768fc7\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " Oct 30 09:25:31.746000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.744501 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fce2865-c606-45f4-84d0-6691b9768fc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fce2865-c606-45f4-84d0-6691b9768fc7-istiod-ca-cert\") pod \"4fce2865-c606-45f4-84d0-6691b9768fc7\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " Oct 30 09:25:31.746000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.744573 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-istio-data\") pod \"cda8e58c-2364-4edb-be89-36fc16609681\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " Oct 30 09:25:31.746000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.744670 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-workload-socket\") pod \"4fce2865-c606-45f4-84d0-6691b9768fc7\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " Oct 30 09:25:31.746000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.744703 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-istio-envoy\") pod \"cda8e58c-2364-4edb-be89-36fc16609681\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " Oct 30 09:25:31.746000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.744829 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-credential-socket" pod "0317fa91-3439-42b9-9b92-7ff6b5f8c31a" (UID: "0317fa91-3439-42b9-9b92-7ff6b5f8c31a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.746000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.745114 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-data" pod "0317fa91-3439-42b9-9b92-7ff6b5f8c31a" (UID: "0317fa91-3439-42b9-9b92-7ff6b5f8c31a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.746000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.745375 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-istio-data" pod "cda8e58c-2364-4edb-be89-36fc16609681" (UID: "cda8e58c-2364-4edb-be89-36fc16609681"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.746000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.745620 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-workload-socket" pod "4fce2865-c606-45f4-84d0-6691b9768fc7" (UID: "4fce2865-c606-45f4-84d0-6691b9768fc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.746000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.745927 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fce2865-c606-45f4-84d0-6691b9768fc7-istiod-ca-cert" pod "4fce2865-c606-45f4-84d0-6691b9768fc7" (UID: "4fce2865-c606-45f4-84d0-6691b9768fc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:31.746000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.746003 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-credential-socket\") pod \"4fce2865-c606-45f4-84d0-6691b9768fc7\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.746053 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cda8e58c-2364-4edb-be89-36fc16609681-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda8e58c-2364-4edb-be89-36fc16609681-istiod-ca-cert\") pod \"cda8e58c-2364-4edb-be89-36fc16609681\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.746274 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-credential-socket" pod "4fce2865-c606-45f4-84d0-6691b9768fc7" (UID: "4fce2865-c606-45f4-84d0-6691b9768fc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.746290 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cda8e58c-2364-4edb-be89-36fc16609681-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda8e58c-2364-4edb-be89-36fc16609681-istio-podinfo\") pod \"cda8e58c-2364-4edb-be89-36fc16609681\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.746453 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-workload-socket\") pod \"cda8e58c-2364-4edb-be89-36fc16609681\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.746517 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda8e58c-2364-4edb-be89-36fc16609681-istio-token\" (UniqueName: \"kubernetes.io/projected/cda8e58c-2364-4edb-be89-36fc16609681-istio-token\") pod \"cda8e58c-2364-4edb-be89-36fc16609681\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.746566 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda8e58c-2364-4edb-be89-36fc16609681-istiod-ca-cert" pod "cda8e58c-2364-4edb-be89-36fc16609681" (UID: "cda8e58c-2364-4edb-be89-36fc16609681"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.746598 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-workload-certs\") pod \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.746674 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-envoy\") pod \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.746767 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-workload-socket\") pod \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.746826 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-workload-socket" pod "cda8e58c-2364-4edb-be89-36fc16609681" (UID: "cda8e58c-2364-4edb-be89-36fc16609681"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.746825 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istiod-ca-cert\") pod \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.746894 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-envoy\") pod \"4fce2865-c606-45f4-84d0-6691b9768fc7\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.746922 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-token\" (UniqueName: \"kubernetes.io/projected/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-token\") pod \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\" (UID: \"0317fa91-3439-42b9-9b92-7ff6b5f8c31a\") " Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.746953 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-credential-socket\") pod \"cda8e58c-2364-4edb-be89-36fc16609681\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.746981 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-workload-certs\") pod \"4fce2865-c606-45f4-84d0-6691b9768fc7\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.747126 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-token\" (UniqueName: \"kubernetes.io/projected/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-token\") pod \"4fce2865-c606-45f4-84d0-6691b9768fc7\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.747198 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-workload-certs\") pod \"cda8e58c-2364-4edb-be89-36fc16609681\" (UID: \"cda8e58c-2364-4edb-be89-36fc16609681\") " Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.747226 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istiod-ca-cert" pod "0317fa91-3439-42b9-9b92-7ff6b5f8c31a" (UID: "0317fa91-3439-42b9-9b92-7ff6b5f8c31a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.747248 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-data\") pod \"4fce2865-c606-45f4-84d0-6691b9768fc7\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " Oct 30 09:25:31.747464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.747307 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fce2865-c606-45f4-84d0-6691b9768fc7-kube-api-access-fhrpd\" (UniqueName: \"kubernetes.io/projected/4fce2865-c606-45f4-84d0-6691b9768fc7-kube-api-access-fhrpd\") pod \"4fce2865-c606-45f4-84d0-6691b9768fc7\" (UID: \"4fce2865-c606-45f4-84d0-6691b9768fc7\") " Oct 30 09:25:31.749017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.748679 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.749017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.748713 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.749017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.748714 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-credential-socket" pod "cda8e58c-2364-4edb-be89-36fc16609681" (UID: "cda8e58c-2364-4edb-be89-36fc16609681"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.749017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.748729 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fce2865-c606-45f4-84d0-6691b9768fc7-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.749017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.748866 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.749017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.748882 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.749017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.748906 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.749017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.748922 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda8e58c-2364-4edb-be89-36fc16609681-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.749017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.748936 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.749017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.748949 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.752317 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.752279 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-workload-certs" pod "0317fa91-3439-42b9-9b92-7ff6b5f8c31a" (UID: "0317fa91-3439-42b9-9b92-7ff6b5f8c31a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.752525 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.752471 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-workload-certs" pod "4fce2865-c606-45f4-84d0-6691b9768fc7" (UID: "4fce2865-c606-45f4-84d0-6691b9768fc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.752631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.752537 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-workload-socket" pod "0317fa91-3439-42b9-9b92-7ff6b5f8c31a" (UID: "0317fa91-3439-42b9-9b92-7ff6b5f8c31a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.752856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.752820 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-workload-certs" pod "cda8e58c-2364-4edb-be89-36fc16609681" (UID: "cda8e58c-2364-4edb-be89-36fc16609681"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.753042 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.753021 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-data" pod "4fce2865-c606-45f4-84d0-6691b9768fc7" (UID: "4fce2865-c606-45f4-84d0-6691b9768fc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.754018 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.753979 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-kube-api-access-6zrc8" pod "0317fa91-3439-42b9-9b92-7ff6b5f8c31a" (UID: "0317fa91-3439-42b9-9b92-7ff6b5f8c31a"). InnerVolumeSpecName "kube-api-access-6zrc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:31.756125 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.756091 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-token" pod "4fce2865-c606-45f4-84d0-6691b9768fc7" (UID: "4fce2865-c606-45f4-84d0-6691b9768fc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:31.756680 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.756655 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-istio-envoy" pod "cda8e58c-2364-4edb-be89-36fc16609681" (UID: "cda8e58c-2364-4edb-be89-36fc16609681"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.756781 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.756726 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-podinfo" pod "4fce2865-c606-45f4-84d0-6691b9768fc7" (UID: "4fce2865-c606-45f4-84d0-6691b9768fc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:31.756841 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.756818 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda8e58c-2364-4edb-be89-36fc16609681-istio-token" pod "cda8e58c-2364-4edb-be89-36fc16609681" (UID: "cda8e58c-2364-4edb-be89-36fc16609681"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:31.756957 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.756933 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-podinfo" pod "0317fa91-3439-42b9-9b92-7ff6b5f8c31a" (UID: "0317fa91-3439-42b9-9b92-7ff6b5f8c31a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:31.757011 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.756985 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-envoy" pod "4fce2865-c606-45f4-84d0-6691b9768fc7" (UID: "4fce2865-c606-45f4-84d0-6691b9768fc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.757098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.757044 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-envoy" pod "0317fa91-3439-42b9-9b92-7ff6b5f8c31a" (UID: "0317fa91-3439-42b9-9b92-7ff6b5f8c31a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.757691 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.757653 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda8e58c-2364-4edb-be89-36fc16609681-kube-api-access-fpzwd" pod "cda8e58c-2364-4edb-be89-36fc16609681" (UID: "cda8e58c-2364-4edb-be89-36fc16609681"). InnerVolumeSpecName "kube-api-access-fpzwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:31.758101 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.758067 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cda8e58c-2364-4edb-be89-36fc16609681-istio-podinfo" pod "cda8e58c-2364-4edb-be89-36fc16609681" (UID: "cda8e58c-2364-4edb-be89-36fc16609681"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:31.758392 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.758365 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-token" pod "0317fa91-3439-42b9-9b92-7ff6b5f8c31a" (UID: "0317fa91-3439-42b9-9b92-7ff6b5f8c31a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:31.759417 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.759376 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fce2865-c606-45f4-84d0-6691b9768fc7-kube-api-access-fhrpd" pod "4fce2865-c606-45f4-84d0-6691b9768fc7" (UID: "4fce2865-c606-45f4-84d0-6691b9768fc7"). InnerVolumeSpecName "kube-api-access-fhrpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:31.849662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.849552 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-istio-envoy\") pod \"929a1bc8-672b-49b2-8344-9253178bd699\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " Oct 30 09:25:31.849662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.849621 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-credential-socket\") pod \"929a1bc8-672b-49b2-8344-9253178bd699\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " Oct 30 09:25:31.849870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.849680 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929a1bc8-672b-49b2-8344-9253178bd699-istio-token\" (UniqueName: \"kubernetes.io/projected/929a1bc8-672b-49b2-8344-9253178bd699-istio-token\") pod \"929a1bc8-672b-49b2-8344-9253178bd699\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " Oct 30 09:25:31.849870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.849724 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-workload-socket\") pod \"929a1bc8-672b-49b2-8344-9253178bd699\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " Oct 30 09:25:31.849870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.849768 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/929a1bc8-672b-49b2-8344-9253178bd699-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929a1bc8-672b-49b2-8344-9253178bd699-istiod-ca-cert\") pod \"929a1bc8-672b-49b2-8344-9253178bd699\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " Oct 30 09:25:31.849870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.849830 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/929a1bc8-672b-49b2-8344-9253178bd699-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929a1bc8-672b-49b2-8344-9253178bd699-istio-podinfo\") pod \"929a1bc8-672b-49b2-8344-9253178bd699\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " Oct 30 09:25:31.850064 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.849930 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-istio-data\") pod \"929a1bc8-672b-49b2-8344-9253178bd699\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " Oct 30 09:25:31.850064 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.849964 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-workload-certs\") pod \"929a1bc8-672b-49b2-8344-9253178bd699\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " Oct 30 09:25:31.850064 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.850038 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929a1bc8-672b-49b2-8344-9253178bd699-kube-api-access-dnzcr\" (UniqueName: \"kubernetes.io/projected/929a1bc8-672b-49b2-8344-9253178bd699-kube-api-access-dnzcr\") pod \"929a1bc8-672b-49b2-8344-9253178bd699\" (UID: \"929a1bc8-672b-49b2-8344-9253178bd699\") " Oct 30 09:25:31.850694 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.850658 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-credential-socket" pod "929a1bc8-672b-49b2-8344-9253178bd699" (UID: "929a1bc8-672b-49b2-8344-9253178bd699"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.850694 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.850668 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-workload-socket" pod "929a1bc8-672b-49b2-8344-9253178bd699" (UID: "929a1bc8-672b-49b2-8344-9253178bd699"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.850847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.850795 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.850847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.850818 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda8e58c-2364-4edb-be89-36fc16609681-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.850847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.850832 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.850962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.850846 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda8e58c-2364-4edb-be89-36fc16609681-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.850962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.850864 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.850962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.850876 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.850962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.850889 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.850962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.850898 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-workload-certs" pod "929a1bc8-672b-49b2-8344-9253178bd699" (UID: "929a1bc8-672b-49b2-8344-9253178bd699"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.850962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.850904 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.850962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.850946 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.851262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.850962 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.851262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.850981 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.851262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.850997 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.851262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.851010 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda8e58c-2364-4edb-be89-36fc16609681-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.851262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.851024 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.851262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.851038 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhrpd\" (UniqueName: \"kubernetes.io/projected/4fce2865-c606-45f4-84d0-6691b9768fc7-kube-api-access-fhrpd\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.851262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.851053 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpzwd\" (UniqueName: \"kubernetes.io/projected/cda8e58c-2364-4edb-be89-36fc16609681-kube-api-access-fpzwd\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.851262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.851103 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.851262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.851119 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zrc8\" (UniqueName: \"kubernetes.io/projected/0317fa91-3439-42b9-9b92-7ff6b5f8c31a-kube-api-access-6zrc8\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.851262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.851126 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-istio-data" pod "929a1bc8-672b-49b2-8344-9253178bd699" (UID: "929a1bc8-672b-49b2-8344-9253178bd699"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.851262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.851133 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fce2865-c606-45f4-84d0-6691b9768fc7-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.852019 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.851333 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929a1bc8-672b-49b2-8344-9253178bd699-istiod-ca-cert" pod "929a1bc8-672b-49b2-8344-9253178bd699" (UID: "929a1bc8-672b-49b2-8344-9253178bd699"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:31.852445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.852422 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929a1bc8-672b-49b2-8344-9253178bd699-istio-token" pod "929a1bc8-672b-49b2-8344-9253178bd699" (UID: "929a1bc8-672b-49b2-8344-9253178bd699"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:31.852553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.852444 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-istio-envoy" pod "929a1bc8-672b-49b2-8344-9253178bd699" (UID: "929a1bc8-672b-49b2-8344-9253178bd699"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.853411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.853395 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929a1bc8-672b-49b2-8344-9253178bd699-kube-api-access-dnzcr" pod "929a1bc8-672b-49b2-8344-9253178bd699" (UID: "929a1bc8-672b-49b2-8344-9253178bd699"). InnerVolumeSpecName "kube-api-access-dnzcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:31.853946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.853926 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/929a1bc8-672b-49b2-8344-9253178bd699-istio-podinfo" pod "929a1bc8-672b-49b2-8344-9253178bd699" (UID: "929a1bc8-672b-49b2-8344-9253178bd699"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:31.952420 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.952384 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.952420 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.952413 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.952420 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.952423 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnzcr\" (UniqueName: \"kubernetes.io/projected/929a1bc8-672b-49b2-8344-9253178bd699-kube-api-access-dnzcr\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.952696 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.952435 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.952696 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.952444 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929a1bc8-672b-49b2-8344-9253178bd699-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.952696 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.952454 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929a1bc8-672b-49b2-8344-9253178bd699-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.952696 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.952462 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929a1bc8-672b-49b2-8344-9253178bd699-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.952696 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:31.952471 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929a1bc8-672b-49b2-8344-9253178bd699-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:32.124071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.124032 5938 generic.go:356] "Generic (PLEG): container finished" podID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerID="f0569446e9f952d35b24f776b142855f8169a78697abd78b1f64a2e4737c8ed3" exitCode=0 Oct 30 09:25:32.124286 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.124087 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" Oct 30 09:25:32.124286 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.124121 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" event={"ID":"0317fa91-3439-42b9-9b92-7ff6b5f8c31a","Type":"ContainerDied","Data":"f0569446e9f952d35b24f776b142855f8169a78697abd78b1f64a2e4737c8ed3"} Oct 30 09:25:32.124286 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.124181 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547" event={"ID":"0317fa91-3439-42b9-9b92-7ff6b5f8c31a","Type":"ContainerDied","Data":"618baf2da57d35de349da2b65149410dd287c83b216913108ecd1cbce0db6136"} Oct 30 09:25:32.124286 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.124196 5938 scope.go:122] "RemoveContainer" containerID="f0569446e9f952d35b24f776b142855f8169a78697abd78b1f64a2e4737c8ed3" Oct 30 09:25:32.125917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.125890 5938 generic.go:356] "Generic (PLEG): container finished" podID="cda8e58c-2364-4edb-be89-36fc16609681" containerID="243f303b17815d116e7f0eac32302a4befd813562dd9217ca1def3fe16ebb129" exitCode=0 Oct 30 09:25:32.126042 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.125953 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" Oct 30 09:25:32.126042 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.126001 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" event={"ID":"cda8e58c-2364-4edb-be89-36fc16609681","Type":"ContainerDied","Data":"243f303b17815d116e7f0eac32302a4befd813562dd9217ca1def3fe16ebb129"} Oct 30 09:25:32.126163 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.126048 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh" event={"ID":"cda8e58c-2364-4edb-be89-36fc16609681","Type":"ContainerDied","Data":"ee15df3fdfe022721aba8ecdf9adb514784fd08099d4d69e01c826c8019a9672"} Oct 30 09:25:32.127784 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.127757 5938 generic.go:356] "Generic (PLEG): container finished" podID="4fce2865-c606-45f4-84d0-6691b9768fc7" containerID="e8cdf4906214880aaabfc38e9a3691a0ac56a57cf4e55858f373b39b6af6f15f" exitCode=0 Oct 30 09:25:32.127925 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.127813 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" Oct 30 09:25:32.127925 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.127847 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" event={"ID":"4fce2865-c606-45f4-84d0-6691b9768fc7","Type":"ContainerDied","Data":"e8cdf4906214880aaabfc38e9a3691a0ac56a57cf4e55858f373b39b6af6f15f"} Oct 30 09:25:32.127925 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.127918 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4" event={"ID":"4fce2865-c606-45f4-84d0-6691b9768fc7","Type":"ContainerDied","Data":"60a9203e12c325cf30283013bdb4c7c2841e0776e880dbe9a38ee6f127bf19f4"} Oct 30 09:25:32.129723 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.129697 5938 generic.go:356] "Generic (PLEG): container finished" podID="929a1bc8-672b-49b2-8344-9253178bd699" containerID="0e9c7d5f552a1be49b2ea07326a99317b554c810cc2654b3e766994d73e4267d" exitCode=0 Oct 30 09:25:32.129848 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.129748 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" Oct 30 09:25:32.129848 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.129773 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" event={"ID":"929a1bc8-672b-49b2-8344-9253178bd699","Type":"ContainerDied","Data":"0e9c7d5f552a1be49b2ea07326a99317b554c810cc2654b3e766994d73e4267d"} Oct 30 09:25:32.129848 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.129808 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc" event={"ID":"929a1bc8-672b-49b2-8344-9253178bd699","Type":"ContainerDied","Data":"1ecc0290fbc970579859cabc537012dbec44ab764ed87421772c6833ee515989"} Oct 30 09:25:32.138874 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.138197 5938 scope.go:122] "RemoveContainer" containerID="f0569446e9f952d35b24f776b142855f8169a78697abd78b1f64a2e4737c8ed3" Oct 30 09:25:32.141475 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:25:32.141268 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0569446e9f952d35b24f776b142855f8169a78697abd78b1f64a2e4737c8ed3\": container with ID starting with f0569446e9f952d35b24f776b142855f8169a78697abd78b1f64a2e4737c8ed3 not found: ID does not exist" containerID="f0569446e9f952d35b24f776b142855f8169a78697abd78b1f64a2e4737c8ed3" Oct 30 09:25:32.141657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.141489 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0569446e9f952d35b24f776b142855f8169a78697abd78b1f64a2e4737c8ed3"} err="failed to get container status \"f0569446e9f952d35b24f776b142855f8169a78697abd78b1f64a2e4737c8ed3\": rpc error: code = NotFound desc = could not find container \"f0569446e9f952d35b24f776b142855f8169a78697abd78b1f64a2e4737c8ed3\": container with ID starting with f0569446e9f952d35b24f776b142855f8169a78697abd78b1f64a2e4737c8ed3 not found: ID does not exist" Oct 30 09:25:32.141657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.141521 5938 scope.go:122] "RemoveContainer" containerID="243f303b17815d116e7f0eac32302a4befd813562dd9217ca1def3fe16ebb129" Oct 30 09:25:32.158504 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.158481 5938 scope.go:122] "RemoveContainer" containerID="243f303b17815d116e7f0eac32302a4befd813562dd9217ca1def3fe16ebb129" Oct 30 09:25:32.158927 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:25:32.158897 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"243f303b17815d116e7f0eac32302a4befd813562dd9217ca1def3fe16ebb129\": container with ID starting with 243f303b17815d116e7f0eac32302a4befd813562dd9217ca1def3fe16ebb129 not found: ID does not exist" containerID="243f303b17815d116e7f0eac32302a4befd813562dd9217ca1def3fe16ebb129" Oct 30 09:25:32.159029 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.158939 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"243f303b17815d116e7f0eac32302a4befd813562dd9217ca1def3fe16ebb129"} err="failed to get container status \"243f303b17815d116e7f0eac32302a4befd813562dd9217ca1def3fe16ebb129\": rpc error: code = NotFound desc = could not find container \"243f303b17815d116e7f0eac32302a4befd813562dd9217ca1def3fe16ebb129\": container with ID starting with 243f303b17815d116e7f0eac32302a4befd813562dd9217ca1def3fe16ebb129 not found: ID does not exist" Oct 30 09:25:32.159029 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.158964 5938 scope.go:122] "RemoveContainer" containerID="e8cdf4906214880aaabfc38e9a3691a0ac56a57cf4e55858f373b39b6af6f15f" Oct 30 09:25:32.169622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.169574 5938 scope.go:122] "RemoveContainer" containerID="e8cdf4906214880aaabfc38e9a3691a0ac56a57cf4e55858f373b39b6af6f15f" Oct 30 09:25:32.169961 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:25:32.169940 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8cdf4906214880aaabfc38e9a3691a0ac56a57cf4e55858f373b39b6af6f15f\": container with ID starting with e8cdf4906214880aaabfc38e9a3691a0ac56a57cf4e55858f373b39b6af6f15f not found: ID does not exist" containerID="e8cdf4906214880aaabfc38e9a3691a0ac56a57cf4e55858f373b39b6af6f15f" Oct 30 09:25:32.170014 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.169972 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8cdf4906214880aaabfc38e9a3691a0ac56a57cf4e55858f373b39b6af6f15f"} err="failed to get container status \"e8cdf4906214880aaabfc38e9a3691a0ac56a57cf4e55858f373b39b6af6f15f\": rpc error: code = NotFound desc = could not find container \"e8cdf4906214880aaabfc38e9a3691a0ac56a57cf4e55858f373b39b6af6f15f\": container with ID starting with e8cdf4906214880aaabfc38e9a3691a0ac56a57cf4e55858f373b39b6af6f15f not found: ID does not exist" Oct 30 09:25:32.170014 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.169994 5938 scope.go:122] "RemoveContainer" containerID="0e9c7d5f552a1be49b2ea07326a99317b554c810cc2654b3e766994d73e4267d" Oct 30 09:25:32.180554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.180534 5938 scope.go:122] "RemoveContainer" containerID="0e9c7d5f552a1be49b2ea07326a99317b554c810cc2654b3e766994d73e4267d" Oct 30 09:25:32.180934 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:25:32.180912 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e9c7d5f552a1be49b2ea07326a99317b554c810cc2654b3e766994d73e4267d\": container with ID starting with 0e9c7d5f552a1be49b2ea07326a99317b554c810cc2654b3e766994d73e4267d not found: ID does not exist" containerID="0e9c7d5f552a1be49b2ea07326a99317b554c810cc2654b3e766994d73e4267d" Oct 30 09:25:32.180987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.180943 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e9c7d5f552a1be49b2ea07326a99317b554c810cc2654b3e766994d73e4267d"} err="failed to get container status \"0e9c7d5f552a1be49b2ea07326a99317b554c810cc2654b3e766994d73e4267d\": rpc error: code = NotFound desc = could not find container \"0e9c7d5f552a1be49b2ea07326a99317b554c810cc2654b3e766994d73e4267d\": container with ID starting with 0e9c7d5f552a1be49b2ea07326a99317b554c810cc2654b3e766994d73e4267d not found: ID does not exist" Oct 30 09:25:32.202251 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.202219 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547"] Oct 30 09:25:32.215330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.215294 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-7s547"] Oct 30 09:25:32.260536 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.260505 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" path="/var/lib/kubelet/pods/0317fa91-3439-42b9-9b92-7ff6b5f8c31a/volumes" Oct 30 09:25:32.279147 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.279097 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4"] Oct 30 09:25:32.281949 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.281918 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-dtbb4"] Oct 30 09:25:32.332244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.332191 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc"] Oct 30 09:25:32.347436 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.347396 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-pc2gc"] Oct 30 09:25:32.398808 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.398702 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh"] Oct 30 09:25:32.405136 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:32.405095 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-zbdjh"] Oct 30 09:25:34.257699 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:34.257664 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fce2865-c606-45f4-84d0-6691b9768fc7" path="/var/lib/kubelet/pods/4fce2865-c606-45f4-84d0-6691b9768fc7/volumes" Oct 30 09:25:34.258197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:34.258181 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929a1bc8-672b-49b2-8344-9253178bd699" path="/var/lib/kubelet/pods/929a1bc8-672b-49b2-8344-9253178bd699/volumes" Oct 30 09:25:34.258641 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:34.258627 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cda8e58c-2364-4edb-be89-36fc16609681" path="/var/lib/kubelet/pods/cda8e58c-2364-4edb-be89-36fc16609681/volumes" Oct 30 09:25:34.889473 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:34.889346 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc"] Oct 30 09:25:34.889746 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:34.889652 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" podUID="dfa1d623-e9d5-4c67-ad15-1fb90536fc46" containerName="istio-proxy" containerID="cri-o://72cfc67f3b9ff07cc11fcb33ce68645cd6352d0eceadc98b5c360f0eb173c219" gracePeriod=30 Oct 30 09:25:40.197736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.197626 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" event={"ID":"dfa1d623-e9d5-4c67-ad15-1fb90536fc46","Type":"ContainerDied","Data":"72cfc67f3b9ff07cc11fcb33ce68645cd6352d0eceadc98b5c360f0eb173c219"} Oct 30 09:25:40.197736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.197640 5938 generic.go:356] "Generic (PLEG): container finished" podID="dfa1d623-e9d5-4c67-ad15-1fb90536fc46" containerID="72cfc67f3b9ff07cc11fcb33ce68645cd6352d0eceadc98b5c360f0eb173c219" exitCode=0 Oct 30 09:25:40.244640 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.244578 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:25:40.327434 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.327029 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-data\") pod \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " Oct 30 09:25:40.327900 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.327859 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-token\" (UniqueName: \"kubernetes.io/projected/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-token\") pod \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " Oct 30 09:25:40.328494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.328462 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-podinfo\") pod \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " Oct 30 09:25:40.329133 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.329097 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-workload-certs\") pod \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " Oct 30 09:25:40.329781 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.329356 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-kube-api-access-qs7w4\" (UniqueName: \"kubernetes.io/projected/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-kube-api-access-qs7w4\") pod \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " Oct 30 09:25:40.329996 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.329965 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-credential-socket\") pod \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " Oct 30 09:25:40.344669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.342806 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-data" pod "dfa1d623-e9d5-4c67-ad15-1fb90536fc46" (UID: "dfa1d623-e9d5-4c67-ad15-1fb90536fc46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:40.361468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.346007 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-workload-certs" pod "dfa1d623-e9d5-4c67-ad15-1fb90536fc46" (UID: "dfa1d623-e9d5-4c67-ad15-1fb90536fc46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:40.361468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.347542 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-token" pod "dfa1d623-e9d5-4c67-ad15-1fb90536fc46" (UID: "dfa1d623-e9d5-4c67-ad15-1fb90536fc46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:40.361468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.347638 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-workload-socket\") pod \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " Oct 30 09:25:40.361468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.347749 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-envoy\") pod \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " Oct 30 09:25:40.361468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.347828 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istiod-ca-cert\") pod \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\" (UID: \"dfa1d623-e9d5-4c67-ad15-1fb90536fc46\") " Oct 30 09:25:40.361468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.349021 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.361468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.349059 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.361468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.349074 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.361468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.349490 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istiod-ca-cert" pod "dfa1d623-e9d5-4c67-ad15-1fb90536fc46" (UID: "dfa1d623-e9d5-4c67-ad15-1fb90536fc46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:40.361468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.349776 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-workload-socket" pod "dfa1d623-e9d5-4c67-ad15-1fb90536fc46" (UID: "dfa1d623-e9d5-4c67-ad15-1fb90536fc46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:40.377727 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.362246 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-podinfo" pod "dfa1d623-e9d5-4c67-ad15-1fb90536fc46" (UID: "dfa1d623-e9d5-4c67-ad15-1fb90536fc46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:40.390390 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.383336 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-envoy" pod "dfa1d623-e9d5-4c67-ad15-1fb90536fc46" (UID: "dfa1d623-e9d5-4c67-ad15-1fb90536fc46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:40.390390 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.390261 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-kube-api-access-qs7w4" pod "dfa1d623-e9d5-4c67-ad15-1fb90536fc46" (UID: "dfa1d623-e9d5-4c67-ad15-1fb90536fc46"). InnerVolumeSpecName "kube-api-access-qs7w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:40.395821 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.394852 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-credential-socket" pod "dfa1d623-e9d5-4c67-ad15-1fb90536fc46" (UID: "dfa1d623-e9d5-4c67-ad15-1fb90536fc46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:40.451485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.451381 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs7w4\" (UniqueName: \"kubernetes.io/projected/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-kube-api-access-qs7w4\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.451485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.451422 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.451485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.451436 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.451485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.451450 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.451485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.451462 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.451485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:40.451473 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa1d623-e9d5-4c67-ad15-1fb90536fc46-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:41.207379 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:41.207286 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" Oct 30 09:25:41.207379 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:41.207305 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc" event={"ID":"dfa1d623-e9d5-4c67-ad15-1fb90536fc46","Type":"ContainerDied","Data":"ad8a25c87664ca0432ba832793ac29cce0fc62603a6c912cdc674d0b9f051d9a"} Oct 30 09:25:41.207896 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:41.207423 5938 scope.go:122] "RemoveContainer" containerID="72cfc67f3b9ff07cc11fcb33ce68645cd6352d0eceadc98b5c360f0eb173c219" Oct 30 09:25:41.281857 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:41.281817 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc"] Oct 30 09:25:41.285874 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:41.285820 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-d6gzc"] Oct 30 09:25:42.258510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:42.258461 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfa1d623-e9d5-4c67-ad15-1fb90536fc46" path="/var/lib/kubelet/pods/dfa1d623-e9d5-4c67-ad15-1fb90536fc46/volumes" Oct 30 09:25:42.738409 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:42.738365 5938 scope.go:122] "RemoveContainer" containerID="8eafd18cee6cae022e8b3d2bb75979fee4429aabf486a6971296a320fc4fd959" Oct 30 09:25:45.813974 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:25:45.813935 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:25:52.091543 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:52.091499 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt"] Oct 30 09:25:53.441514 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:53.441479 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc"] Oct 30 09:25:53.442069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:53.441749 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" podUID="946ef50b-d7e0-4b57-a1f3-d91120bee9e5" containerName="maas-api" containerID="cri-o://86ebb130e50e829a1f210e34629e90b707a90036f68f2ca2495dbc4e84ece38b" gracePeriod=30 Oct 30 09:25:53.770645 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:53.770617 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:25:53.908796 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:53.908757 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-kube-api-access-npmwq\" (UniqueName: \"kubernetes.io/projected/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-kube-api-access-npmwq\") pod \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\" (UID: \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\") " Oct 30 09:25:53.908977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:53.908834 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-maas-api-tls\") pod \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\" (UID: \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\") " Oct 30 09:25:53.908977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:53.908939 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-openshift-service-ca\") pod \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\" (UID: \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\") " Oct 30 09:25:53.909055 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:53.909032 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-metrics-certs\") pod \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\" (UID: \"946ef50b-d7e0-4b57-a1f3-d91120bee9e5\") " Oct 30 09:25:53.909372 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:53.909339 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-openshift-service-ca" pod "946ef50b-d7e0-4b57-a1f3-d91120bee9e5" (UID: "946ef50b-d7e0-4b57-a1f3-d91120bee9e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:53.911276 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:53.911233 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-maas-api-tls" pod "946ef50b-d7e0-4b57-a1f3-d91120bee9e5" (UID: "946ef50b-d7e0-4b57-a1f3-d91120bee9e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:25:53.911276 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:53.911252 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-metrics-certs" pod "946ef50b-d7e0-4b57-a1f3-d91120bee9e5" (UID: "946ef50b-d7e0-4b57-a1f3-d91120bee9e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:25:53.911446 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:53.911284 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-kube-api-access-npmwq" pod "946ef50b-d7e0-4b57-a1f3-d91120bee9e5" (UID: "946ef50b-d7e0-4b57-a1f3-d91120bee9e5"). InnerVolumeSpecName "kube-api-access-npmwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:54.011647 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:54.011473 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-metrics-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.011647 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:54.011513 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npmwq\" (UniqueName: \"kubernetes.io/projected/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-kube-api-access-npmwq\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.011647 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:54.011528 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-maas-api-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.011647 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:54.011541 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/946ef50b-d7e0-4b57-a1f3-d91120bee9e5-openshift-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.311579 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:54.311538 5938 generic.go:356] "Generic (PLEG): container finished" podID="946ef50b-d7e0-4b57-a1f3-d91120bee9e5" containerID="86ebb130e50e829a1f210e34629e90b707a90036f68f2ca2495dbc4e84ece38b" exitCode=0 Oct 30 09:25:54.311838 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:54.311639 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" Oct 30 09:25:54.311838 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:54.311629 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" event={"ID":"946ef50b-d7e0-4b57-a1f3-d91120bee9e5","Type":"ContainerDied","Data":"86ebb130e50e829a1f210e34629e90b707a90036f68f2ca2495dbc4e84ece38b"} Oct 30 09:25:54.311838 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:54.311708 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc" event={"ID":"946ef50b-d7e0-4b57-a1f3-d91120bee9e5","Type":"ContainerDied","Data":"ea7def1be610f80348a0f449a92607a5b5034e83a29a37d5bc0515ece95c99ed"} Oct 30 09:25:54.311838 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:54.311740 5938 scope.go:122] "RemoveContainer" containerID="86ebb130e50e829a1f210e34629e90b707a90036f68f2ca2495dbc4e84ece38b" Oct 30 09:25:54.325500 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:54.325477 5938 scope.go:122] "RemoveContainer" containerID="86ebb130e50e829a1f210e34629e90b707a90036f68f2ca2495dbc4e84ece38b" Oct 30 09:25:54.325954 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:25:54.325921 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ebb130e50e829a1f210e34629e90b707a90036f68f2ca2495dbc4e84ece38b\": container with ID starting with 86ebb130e50e829a1f210e34629e90b707a90036f68f2ca2495dbc4e84ece38b not found: ID does not exist" containerID="86ebb130e50e829a1f210e34629e90b707a90036f68f2ca2495dbc4e84ece38b" Oct 30 09:25:54.326095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:54.325968 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ebb130e50e829a1f210e34629e90b707a90036f68f2ca2495dbc4e84ece38b"} err="failed to get container status \"86ebb130e50e829a1f210e34629e90b707a90036f68f2ca2495dbc4e84ece38b\": rpc error: code = NotFound desc = could not find container \"86ebb130e50e829a1f210e34629e90b707a90036f68f2ca2495dbc4e84ece38b\": container with ID starting with 86ebb130e50e829a1f210e34629e90b707a90036f68f2ca2495dbc4e84ece38b not found: ID does not exist" Oct 30 09:25:54.367837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:54.367791 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc"] Oct 30 09:25:54.381143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:54.381108 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbe974-58fc797dfd-lrpjc"] Oct 30 09:25:56.257227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:56.257191 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946ef50b-d7e0-4b57-a1f3-d91120bee9e5" path="/var/lib/kubelet/pods/946ef50b-d7e0-4b57-a1f3-d91120bee9e5/volumes" Oct 30 09:25:57.092538 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:57.092492 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" podUID="abb08123-121f-4432-93fb-c4b7a5f8bb6a" containerName="payload-pre-processing" containerID="cri-o://eb0108bbc44706e727090725bf3acad3c344ca011e4b85977f52ea6e4b795ae1" gracePeriod=25 Oct 30 09:25:57.335622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:57.335568 5938 generic.go:356] "Generic (PLEG): container finished" podID="abb08123-121f-4432-93fb-c4b7a5f8bb6a" containerID="eb0108bbc44706e727090725bf3acad3c344ca011e4b85977f52ea6e4b795ae1" exitCode=0 Oct 30 09:25:57.335998 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:57.335622 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" event={"ID":"abb08123-121f-4432-93fb-c4b7a5f8bb6a","Type":"ContainerDied","Data":"eb0108bbc44706e727090725bf3acad3c344ca011e4b85977f52ea6e4b795ae1"} Oct 30 09:25:57.366835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:57.366804 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" Oct 30 09:25:57.458772 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:57.458736 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abb08123-121f-4432-93fb-c4b7a5f8bb6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb08123-121f-4432-93fb-c4b7a5f8bb6a-plugins-config-volume\") pod \"abb08123-121f-4432-93fb-c4b7a5f8bb6a\" (UID: \"abb08123-121f-4432-93fb-c4b7a5f8bb6a\") " Oct 30 09:25:57.458959 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:57.458841 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb08123-121f-4432-93fb-c4b7a5f8bb6a-kube-api-access-fm25n\" (UniqueName: \"kubernetes.io/projected/abb08123-121f-4432-93fb-c4b7a5f8bb6a-kube-api-access-fm25n\") pod \"abb08123-121f-4432-93fb-c4b7a5f8bb6a\" (UID: \"abb08123-121f-4432-93fb-c4b7a5f8bb6a\") " Oct 30 09:25:57.459237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:57.459215 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb08123-121f-4432-93fb-c4b7a5f8bb6a-plugins-config-volume" pod "abb08123-121f-4432-93fb-c4b7a5f8bb6a" (UID: "abb08123-121f-4432-93fb-c4b7a5f8bb6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:57.459401 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:57.459387 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb08123-121f-4432-93fb-c4b7a5f8bb6a-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:57.461151 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:57.461125 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb08123-121f-4432-93fb-c4b7a5f8bb6a-kube-api-access-fm25n" pod "abb08123-121f-4432-93fb-c4b7a5f8bb6a" (UID: "abb08123-121f-4432-93fb-c4b7a5f8bb6a"). InnerVolumeSpecName "kube-api-access-fm25n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:57.560298 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:57.560266 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm25n\" (UniqueName: \"kubernetes.io/projected/abb08123-121f-4432-93fb-c4b7a5f8bb6a-kube-api-access-fm25n\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:25:58.344714 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:58.344684 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" Oct 30 09:25:58.344714 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:58.344690 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt" event={"ID":"abb08123-121f-4432-93fb-c4b7a5f8bb6a","Type":"ContainerDied","Data":"8dbe6d1b31db75367ddb914247564bf9dd2ac7469f2d384cfdefca127d547a08"} Oct 30 09:25:58.345233 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:58.344750 5938 scope.go:122] "RemoveContainer" containerID="eb0108bbc44706e727090725bf3acad3c344ca011e4b85977f52ea6e4b795ae1" Oct 30 09:25:58.400760 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:58.400720 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt"] Oct 30 09:25:58.414568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:25:58.414536 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9371b7-7c86bbc6c-mq9mt"] Oct 30 09:26:00.257819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:00.257785 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb08123-121f-4432-93fb-c4b7a5f8bb6a" path="/var/lib/kubelet/pods/abb08123-121f-4432-93fb-c4b7a5f8bb6a/volumes" Oct 30 09:26:05.130343 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.130296 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw"] Oct 30 09:26:05.130933 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.130569 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" podUID="e64333ed-ee2e-4099-82ca-6ae5f140058f" containerName="maas-api" containerID="cri-o://5ac1c42b48ffe4c0352119e18b174b0f6aaa942d16c024e899848b1a18382d5a" gracePeriod=30 Oct 30 09:26:05.413370 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.413163 5938 generic.go:356] "Generic (PLEG): container finished" podID="e64333ed-ee2e-4099-82ca-6ae5f140058f" containerID="5ac1c42b48ffe4c0352119e18b174b0f6aaa942d16c024e899848b1a18382d5a" exitCode=0 Oct 30 09:26:05.413370 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.413270 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" event={"ID":"e64333ed-ee2e-4099-82ca-6ae5f140058f","Type":"ContainerDied","Data":"5ac1c42b48ffe4c0352119e18b174b0f6aaa942d16c024e899848b1a18382d5a"} Oct 30 09:26:05.497239 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.497192 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:26:05.546856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.546819 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64333ed-ee2e-4099-82ca-6ae5f140058f-kube-api-access-m44tw\" (UniqueName: \"kubernetes.io/projected/e64333ed-ee2e-4099-82ca-6ae5f140058f-kube-api-access-m44tw\") pod \"e64333ed-ee2e-4099-82ca-6ae5f140058f\" (UID: \"e64333ed-ee2e-4099-82ca-6ae5f140058f\") " Oct 30 09:26:05.547211 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.547189 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64333ed-ee2e-4099-82ca-6ae5f140058f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64333ed-ee2e-4099-82ca-6ae5f140058f-metrics-certs\") pod \"e64333ed-ee2e-4099-82ca-6ae5f140058f\" (UID: \"e64333ed-ee2e-4099-82ca-6ae5f140058f\") " Oct 30 09:26:05.547417 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.547389 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64333ed-ee2e-4099-82ca-6ae5f140058f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e64333ed-ee2e-4099-82ca-6ae5f140058f-maas-api-tls\") pod \"e64333ed-ee2e-4099-82ca-6ae5f140058f\" (UID: \"e64333ed-ee2e-4099-82ca-6ae5f140058f\") " Oct 30 09:26:05.547668 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.547645 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64333ed-ee2e-4099-82ca-6ae5f140058f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64333ed-ee2e-4099-82ca-6ae5f140058f-openshift-service-ca\") pod \"e64333ed-ee2e-4099-82ca-6ae5f140058f\" (UID: \"e64333ed-ee2e-4099-82ca-6ae5f140058f\") " Oct 30 09:26:05.550815 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.550780 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64333ed-ee2e-4099-82ca-6ae5f140058f-openshift-service-ca" pod "e64333ed-ee2e-4099-82ca-6ae5f140058f" (UID: "e64333ed-ee2e-4099-82ca-6ae5f140058f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:05.552855 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.552799 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64333ed-ee2e-4099-82ca-6ae5f140058f-metrics-certs" pod "e64333ed-ee2e-4099-82ca-6ae5f140058f" (UID: "e64333ed-ee2e-4099-82ca-6ae5f140058f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:26:05.553179 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.553150 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64333ed-ee2e-4099-82ca-6ae5f140058f-maas-api-tls" pod "e64333ed-ee2e-4099-82ca-6ae5f140058f" (UID: "e64333ed-ee2e-4099-82ca-6ae5f140058f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:26:05.583797 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.573264 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64333ed-ee2e-4099-82ca-6ae5f140058f-kube-api-access-m44tw" pod "e64333ed-ee2e-4099-82ca-6ae5f140058f" (UID: "e64333ed-ee2e-4099-82ca-6ae5f140058f"). InnerVolumeSpecName "kube-api-access-m44tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:05.664465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.663991 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m44tw\" (UniqueName: \"kubernetes.io/projected/e64333ed-ee2e-4099-82ca-6ae5f140058f-kube-api-access-m44tw\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:05.664465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.664025 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64333ed-ee2e-4099-82ca-6ae5f140058f-metrics-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:05.664465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.664043 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e64333ed-ee2e-4099-82ca-6ae5f140058f-maas-api-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:05.664465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:05.664056 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64333ed-ee2e-4099-82ca-6ae5f140058f-openshift-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:06.421386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:06.421343 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" event={"ID":"e64333ed-ee2e-4099-82ca-6ae5f140058f","Type":"ContainerDied","Data":"81fe6d4f8474cc1a5db300c919c2adff790ae64e6aa50abcb18e76d40263bc8c"} Oct 30 09:26:06.421860 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:06.421407 5938 scope.go:122] "RemoveContainer" containerID="5ac1c42b48ffe4c0352119e18b174b0f6aaa942d16c024e899848b1a18382d5a" Oct 30 09:26:06.421860 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:06.421359 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw" Oct 30 09:26:06.482072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:06.482030 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw"] Oct 30 09:26:06.494096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:06.494060 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-962e2e-5dd79564b5-cjtlw"] Oct 30 09:26:08.258056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:08.258015 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64333ed-ee2e-4099-82ca-6ae5f140058f" path="/var/lib/kubelet/pods/e64333ed-ee2e-4099-82ca-6ae5f140058f/volumes" Oct 30 09:26:15.304825 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.304782 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2"] Oct 30 09:26:15.305680 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.305645 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" podUID="cb18f58e-ec28-4e52-a9c0-f8bee67cf18a" containerName="maas-api" containerID="cri-o://e0ca2787b2f6d867f97d65a677af3a77624f389749f99dc39220e6c6ea16e14a" gracePeriod=30 Oct 30 09:26:15.492421 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.491995 5938 generic.go:356] "Generic (PLEG): container finished" podID="cb18f58e-ec28-4e52-a9c0-f8bee67cf18a" containerID="e0ca2787b2f6d867f97d65a677af3a77624f389749f99dc39220e6c6ea16e14a" exitCode=0 Oct 30 09:26:15.492421 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.492235 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" event={"ID":"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a","Type":"ContainerDied","Data":"e0ca2787b2f6d867f97d65a677af3a77624f389749f99dc39220e6c6ea16e14a"} Oct 30 09:26:15.687988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.687956 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:26:15.824199 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:26:15.824167 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:26:15.829176 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.829148 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-kube-api-access-cs776\" (UniqueName: \"kubernetes.io/projected/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-kube-api-access-cs776\") pod \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\" (UID: \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\") " Oct 30 09:26:15.829399 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.829186 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-maas-api-tls\") pod \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\" (UID: \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\") " Oct 30 09:26:15.829399 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.829329 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-openshift-service-ca\") pod \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\" (UID: \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\") " Oct 30 09:26:15.829399 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.829393 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-metrics-certs\") pod \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\" (UID: \"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a\") " Oct 30 09:26:15.829834 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.829750 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-openshift-service-ca" pod "cb18f58e-ec28-4e52-a9c0-f8bee67cf18a" (UID: "cb18f58e-ec28-4e52-a9c0-f8bee67cf18a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:15.830236 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.830157 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-openshift-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:15.831834 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.831778 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-kube-api-access-cs776" pod "cb18f58e-ec28-4e52-a9c0-f8bee67cf18a" (UID: "cb18f58e-ec28-4e52-a9c0-f8bee67cf18a"). InnerVolumeSpecName "kube-api-access-cs776". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:15.831952 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.831924 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-metrics-certs" pod "cb18f58e-ec28-4e52-a9c0-f8bee67cf18a" (UID: "cb18f58e-ec28-4e52-a9c0-f8bee67cf18a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:26:15.832056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.832024 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-maas-api-tls" pod "cb18f58e-ec28-4e52-a9c0-f8bee67cf18a" (UID: "cb18f58e-ec28-4e52-a9c0-f8bee67cf18a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:26:15.933926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.933728 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-metrics-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:15.933926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.933770 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs776\" (UniqueName: \"kubernetes.io/projected/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-kube-api-access-cs776\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:15.933926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:15.933786 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a-maas-api-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:16.500123 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:16.500092 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" Oct 30 09:26:16.500632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:16.500092 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2" event={"ID":"cb18f58e-ec28-4e52-a9c0-f8bee67cf18a","Type":"ContainerDied","Data":"bc614316b32f292c39d1e33bf2728baaaf76cb26bb78ba0124061c3674345e13"} Oct 30 09:26:16.500632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:16.500247 5938 scope.go:122] "RemoveContainer" containerID="e0ca2787b2f6d867f97d65a677af3a77624f389749f99dc39220e6c6ea16e14a" Oct 30 09:26:16.560473 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:16.560385 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2"] Oct 30 09:26:16.572545 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:16.572511 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-356553-57c6f48775-zj5r2"] Oct 30 09:26:18.276472 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:18.276434 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb18f58e-ec28-4e52-a9c0-f8bee67cf18a" path="/var/lib/kubelet/pods/cb18f58e-ec28-4e52-a9c0-f8bee67cf18a/volumes" Oct 30 09:26:21.528855 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:21.528816 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x"] Oct 30 09:26:26.529871 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:26.529745 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" podUID="0eea09cb-d491-4c49-8013-000393c05e5f" containerName="payload-pre-processing" containerID="cri-o://7dcf78b820fba9623ddda4004416f1264385e6d80182139d9b169585cc375a4c" gracePeriod=25 Oct 30 09:26:26.836892 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:26.836846 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" Oct 30 09:26:26.928155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:26.928117 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eea09cb-d491-4c49-8013-000393c05e5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eea09cb-d491-4c49-8013-000393c05e5f-plugins-config-volume\") pod \"0eea09cb-d491-4c49-8013-000393c05e5f\" (UID: \"0eea09cb-d491-4c49-8013-000393c05e5f\") " Oct 30 09:26:26.928318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:26.928267 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eea09cb-d491-4c49-8013-000393c05e5f-kube-api-access-jws5n\" (UniqueName: \"kubernetes.io/projected/0eea09cb-d491-4c49-8013-000393c05e5f-kube-api-access-jws5n\") pod \"0eea09cb-d491-4c49-8013-000393c05e5f\" (UID: \"0eea09cb-d491-4c49-8013-000393c05e5f\") " Oct 30 09:26:26.928684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:26.928644 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eea09cb-d491-4c49-8013-000393c05e5f-plugins-config-volume" pod "0eea09cb-d491-4c49-8013-000393c05e5f" (UID: "0eea09cb-d491-4c49-8013-000393c05e5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:26.928810 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:26.928780 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eea09cb-d491-4c49-8013-000393c05e5f-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:26.930706 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:26.930674 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eea09cb-d491-4c49-8013-000393c05e5f-kube-api-access-jws5n" pod "0eea09cb-d491-4c49-8013-000393c05e5f" (UID: "0eea09cb-d491-4c49-8013-000393c05e5f"). InnerVolumeSpecName "kube-api-access-jws5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:27.029716 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:27.029668 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jws5n\" (UniqueName: \"kubernetes.io/projected/0eea09cb-d491-4c49-8013-000393c05e5f-kube-api-access-jws5n\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:27.579694 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:27.579659 5938 generic.go:356] "Generic (PLEG): container finished" podID="0eea09cb-d491-4c49-8013-000393c05e5f" containerID="7dcf78b820fba9623ddda4004416f1264385e6d80182139d9b169585cc375a4c" exitCode=0 Oct 30 09:26:27.580235 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:27.579695 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" event={"ID":"0eea09cb-d491-4c49-8013-000393c05e5f","Type":"ContainerDied","Data":"7dcf78b820fba9623ddda4004416f1264385e6d80182139d9b169585cc375a4c"} Oct 30 09:26:27.580235 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:27.579758 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" event={"ID":"0eea09cb-d491-4c49-8013-000393c05e5f","Type":"ContainerDied","Data":"38d2499d04bd87ed935a6e4ea4bc8f99d5460f027e29c20da2b52cadc905a177"} Oct 30 09:26:27.580235 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:27.579711 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x" Oct 30 09:26:27.580235 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:27.579782 5938 scope.go:122] "RemoveContainer" containerID="7dcf78b820fba9623ddda4004416f1264385e6d80182139d9b169585cc375a4c" Oct 30 09:26:27.590508 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:27.590487 5938 scope.go:122] "RemoveContainer" containerID="7dcf78b820fba9623ddda4004416f1264385e6d80182139d9b169585cc375a4c" Oct 30 09:26:27.590915 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:26:27.590891 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dcf78b820fba9623ddda4004416f1264385e6d80182139d9b169585cc375a4c\": container with ID starting with 7dcf78b820fba9623ddda4004416f1264385e6d80182139d9b169585cc375a4c not found: ID does not exist" containerID="7dcf78b820fba9623ddda4004416f1264385e6d80182139d9b169585cc375a4c" Oct 30 09:26:27.590995 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:27.590927 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dcf78b820fba9623ddda4004416f1264385e6d80182139d9b169585cc375a4c"} err="failed to get container status \"7dcf78b820fba9623ddda4004416f1264385e6d80182139d9b169585cc375a4c\": rpc error: code = NotFound desc = could not find container \"7dcf78b820fba9623ddda4004416f1264385e6d80182139d9b169585cc375a4c\": container with ID starting with 7dcf78b820fba9623ddda4004416f1264385e6d80182139d9b169585cc375a4c not found: ID does not exist" Oct 30 09:26:27.634822 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:27.634779 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x"] Oct 30 09:26:27.648547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:27.648511 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4fc2e2-7b997445f5-l2n9x"] Oct 30 09:26:28.257148 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:28.257108 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eea09cb-d491-4c49-8013-000393c05e5f" path="/var/lib/kubelet/pods/0eea09cb-d491-4c49-8013-000393c05e5f/volumes" Oct 30 09:26:29.428175 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:29.428130 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz"] Oct 30 09:26:29.428794 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:29.428413 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" podUID="58a20fda-9d11-47ca-887d-e1dac99e53a2" containerName="istio-proxy" containerID="cri-o://7e437cafd311955f2b446862532a3472307d24d1bc2d5f23b4b04ce09c0debc0" gracePeriod=30 Oct 30 09:26:34.698783 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.698704 5938 generic.go:356] "Generic (PLEG): container finished" podID="58a20fda-9d11-47ca-887d-e1dac99e53a2" containerID="7e437cafd311955f2b446862532a3472307d24d1bc2d5f23b4b04ce09c0debc0" exitCode=0 Oct 30 09:26:34.699140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.698781 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" event={"ID":"58a20fda-9d11-47ca-887d-e1dac99e53a2","Type":"ContainerDied","Data":"7e437cafd311955f2b446862532a3472307d24d1bc2d5f23b4b04ce09c0debc0"} Oct 30 09:26:34.699140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.698868 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" event={"ID":"58a20fda-9d11-47ca-887d-e1dac99e53a2","Type":"ContainerDied","Data":"b266c118ff3fb25eb576a75280b3986480de809a030941127513be697a7f723b"} Oct 30 09:26:34.699140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.698894 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b266c118ff3fb25eb576a75280b3986480de809a030941127513be697a7f723b" Oct 30 09:26:34.699140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.698912 5938 scope.go:122] "RemoveContainer" containerID="1e7efcb9aded3ee94affc28ff686a36b3e75de2e348dce3a8d2c5ff5be0916fe" Oct 30 09:26:34.704761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.704738 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:26:34.770603 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.770547 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-workload-socket\") pod \"58a20fda-9d11-47ca-887d-e1dac99e53a2\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " Oct 30 09:26:34.770776 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.770615 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-token\" (UniqueName: \"kubernetes.io/projected/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-token\") pod \"58a20fda-9d11-47ca-887d-e1dac99e53a2\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " Oct 30 09:26:34.770776 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.770680 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-credential-socket\") pod \"58a20fda-9d11-47ca-887d-e1dac99e53a2\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " Oct 30 09:26:34.770776 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.770768 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-envoy\") pod \"58a20fda-9d11-47ca-887d-e1dac99e53a2\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " Oct 30 09:26:34.770962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.770803 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a20fda-9d11-47ca-887d-e1dac99e53a2-kube-api-access-wf8q4\" (UniqueName: \"kubernetes.io/projected/58a20fda-9d11-47ca-887d-e1dac99e53a2-kube-api-access-wf8q4\") pod \"58a20fda-9d11-47ca-887d-e1dac99e53a2\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " Oct 30 09:26:34.770962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.770837 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-podinfo\") pod \"58a20fda-9d11-47ca-887d-e1dac99e53a2\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " Oct 30 09:26:34.770962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.770915 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a20fda-9d11-47ca-887d-e1dac99e53a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a20fda-9d11-47ca-887d-e1dac99e53a2-istiod-ca-cert\") pod \"58a20fda-9d11-47ca-887d-e1dac99e53a2\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " Oct 30 09:26:34.770962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.770917 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-workload-socket" pod "58a20fda-9d11-47ca-887d-e1dac99e53a2" (UID: "58a20fda-9d11-47ca-887d-e1dac99e53a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:34.771199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.770969 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-data\") pod \"58a20fda-9d11-47ca-887d-e1dac99e53a2\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " Oct 30 09:26:34.771199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.771046 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-credential-socket" pod "58a20fda-9d11-47ca-887d-e1dac99e53a2" (UID: "58a20fda-9d11-47ca-887d-e1dac99e53a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:34.771199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.771075 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-workload-certs\") pod \"58a20fda-9d11-47ca-887d-e1dac99e53a2\" (UID: \"58a20fda-9d11-47ca-887d-e1dac99e53a2\") " Oct 30 09:26:34.771997 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.771363 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a20fda-9d11-47ca-887d-e1dac99e53a2-istiod-ca-cert" pod "58a20fda-9d11-47ca-887d-e1dac99e53a2" (UID: "58a20fda-9d11-47ca-887d-e1dac99e53a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:34.771997 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.771439 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-data" pod "58a20fda-9d11-47ca-887d-e1dac99e53a2" (UID: "58a20fda-9d11-47ca-887d-e1dac99e53a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:34.771997 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.771665 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-workload-certs" pod "58a20fda-9d11-47ca-887d-e1dac99e53a2" (UID: "58a20fda-9d11-47ca-887d-e1dac99e53a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:34.771997 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.771727 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a20fda-9d11-47ca-887d-e1dac99e53a2-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:34.771997 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.771749 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:34.771997 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.771762 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:34.771997 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.771776 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:34.773924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.773895 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-podinfo" pod "58a20fda-9d11-47ca-887d-e1dac99e53a2" (UID: "58a20fda-9d11-47ca-887d-e1dac99e53a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:26:34.774117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.774097 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-token" pod "58a20fda-9d11-47ca-887d-e1dac99e53a2" (UID: "58a20fda-9d11-47ca-887d-e1dac99e53a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:34.774184 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.774109 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a20fda-9d11-47ca-887d-e1dac99e53a2-kube-api-access-wf8q4" pod "58a20fda-9d11-47ca-887d-e1dac99e53a2" (UID: "58a20fda-9d11-47ca-887d-e1dac99e53a2"). InnerVolumeSpecName "kube-api-access-wf8q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:34.774323 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.774302 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-envoy" pod "58a20fda-9d11-47ca-887d-e1dac99e53a2" (UID: "58a20fda-9d11-47ca-887d-e1dac99e53a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:34.873650 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.873581 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:34.873650 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.873649 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:34.873650 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.873661 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:34.873927 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.873677 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf8q4\" (UniqueName: \"kubernetes.io/projected/58a20fda-9d11-47ca-887d-e1dac99e53a2-kube-api-access-wf8q4\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:34.873927 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:34.873688 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a20fda-9d11-47ca-887d-e1dac99e53a2-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:35.707072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:35.707035 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz" Oct 30 09:26:35.803642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:35.803605 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz"] Oct 30 09:26:35.806224 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:35.806190 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-p66wz"] Oct 30 09:26:36.257615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:36.257545 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a20fda-9d11-47ca-887d-e1dac99e53a2" path="/var/lib/kubelet/pods/58a20fda-9d11-47ca-887d-e1dac99e53a2/volumes" Oct 30 09:26:45.884490 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:26:45.884451 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:26:55.027809 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.027767 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9"] Oct 30 09:26:55.028334 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028319 5938 cpu_manager.go:414] "containerMap: removing container" podUID="929a1bc8-672b-49b2-8344-9253178bd699" containerName="istio-proxy" Oct 30 09:26:55.028445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028337 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929a1bc8-672b-49b2-8344-9253178bd699" containerName="istio-proxy" Oct 30 09:26:55.028445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028360 5938 cpu_manager.go:414] "containerMap: removing container" podUID="cb18f58e-ec28-4e52-a9c0-f8bee67cf18a" containerName="maas-api" Oct 30 09:26:55.028445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028366 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb18f58e-ec28-4e52-a9c0-f8bee67cf18a" containerName="maas-api" Oct 30 09:26:55.028445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028374 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e64333ed-ee2e-4099-82ca-6ae5f140058f" containerName="maas-api" Oct 30 09:26:55.028445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028382 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64333ed-ee2e-4099-82ca-6ae5f140058f" containerName="maas-api" Oct 30 09:26:55.028445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028395 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4fce2865-c606-45f4-84d0-6691b9768fc7" containerName="istio-proxy" Oct 30 09:26:55.028445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028400 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fce2865-c606-45f4-84d0-6691b9768fc7" containerName="istio-proxy" Oct 30 09:26:55.028445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028411 5938 cpu_manager.go:414] "containerMap: removing container" podUID="58a20fda-9d11-47ca-887d-e1dac99e53a2" containerName="istio-proxy" Oct 30 09:26:55.028445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028415 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a20fda-9d11-47ca-887d-e1dac99e53a2" containerName="istio-proxy" Oct 30 09:26:55.028445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028423 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0eea09cb-d491-4c49-8013-000393c05e5f" containerName="payload-pre-processing" Oct 30 09:26:55.028445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028427 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eea09cb-d491-4c49-8013-000393c05e5f" containerName="payload-pre-processing" Oct 30 09:26:55.028445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028436 5938 cpu_manager.go:414] "containerMap: removing container" podUID="cda8e58c-2364-4edb-be89-36fc16609681" containerName="istio-proxy" Oct 30 09:26:55.028445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028440 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cda8e58c-2364-4edb-be89-36fc16609681" containerName="istio-proxy" Oct 30 09:26:55.028445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028446 5938 cpu_manager.go:414] "containerMap: removing container" podUID="dfa1d623-e9d5-4c67-ad15-1fb90536fc46" containerName="istio-proxy" Oct 30 09:26:55.028445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028451 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa1d623-e9d5-4c67-ad15-1fb90536fc46" containerName="istio-proxy" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028463 5938 cpu_manager.go:414] "containerMap: removing container" podUID="946ef50b-d7e0-4b57-a1f3-d91120bee9e5" containerName="maas-api" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028470 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946ef50b-d7e0-4b57-a1f3-d91120bee9e5" containerName="maas-api" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028479 5938 cpu_manager.go:414] "containerMap: removing container" podUID="abb08123-121f-4432-93fb-c4b7a5f8bb6a" containerName="payload-pre-processing" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028484 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb08123-121f-4432-93fb-c4b7a5f8bb6a" containerName="payload-pre-processing" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028502 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" containerName="istio-proxy" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028506 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" containerName="istio-proxy" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028515 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerName="istio-proxy" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028519 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerName="istio-proxy" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028532 5938 cpu_manager.go:414] "containerMap: removing container" podUID="58a20fda-9d11-47ca-887d-e1dac99e53a2" containerName="istio-proxy" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028538 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a20fda-9d11-47ca-887d-e1dac99e53a2" containerName="istio-proxy" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028654 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb18f58e-ec28-4e52-a9c0-f8bee67cf18a" containerName="maas-api" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028663 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64333ed-ee2e-4099-82ca-6ae5f140058f" containerName="maas-api" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028671 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fce2865-c606-45f4-84d0-6691b9768fc7" containerName="istio-proxy" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028677 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a20fda-9d11-47ca-887d-e1dac99e53a2" containerName="istio-proxy" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028683 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eea09cb-d491-4c49-8013-000393c05e5f" containerName="payload-pre-processing" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028689 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="cda8e58c-2364-4edb-be89-36fc16609681" containerName="istio-proxy" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028695 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa1d623-e9d5-4c67-ad15-1fb90536fc46" containerName="istio-proxy" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028701 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="946ef50b-d7e0-4b57-a1f3-d91120bee9e5" containerName="maas-api" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028707 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb08123-121f-4432-93fb-c4b7a5f8bb6a" containerName="payload-pre-processing" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028720 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0ba647-3cd7-49c1-9c5e-6872fd5a0fa1" containerName="istio-proxy" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028727 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0317fa91-3439-42b9-9b92-7ff6b5f8c31a" containerName="istio-proxy" Oct 30 09:26:55.028977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.028734 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="929a1bc8-672b-49b2-8344-9253178bd699" containerName="istio-proxy" Oct 30 09:26:55.035377 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.035343 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9" Oct 30 09:26:55.056160 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.056121 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9"] Oct 30 09:26:55.065681 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.065605 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:26:55.065877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.065781 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:26:55.118926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.118883 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053d4858-2ad7-4f38-90c9-7070638157cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9\" (UID: \"053d4858-2ad7-4f38-90c9-7070638157cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9" Oct 30 09:26:55.119143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.119044 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053d4858-2ad7-4f38-90c9-7070638157cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9\" (UID: \"053d4858-2ad7-4f38-90c9-7070638157cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9" Oct 30 09:26:55.119143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.119091 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw94r\" (UniqueName: \"kubernetes.io/projected/053d4858-2ad7-4f38-90c9-7070638157cd-kube-api-access-lw94r\") pod \"maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9\" (UID: \"053d4858-2ad7-4f38-90c9-7070638157cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9" Oct 30 09:26:55.220330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.220286 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053d4858-2ad7-4f38-90c9-7070638157cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9\" (UID: \"053d4858-2ad7-4f38-90c9-7070638157cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9" Oct 30 09:26:55.220492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.220360 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053d4858-2ad7-4f38-90c9-7070638157cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9\" (UID: \"053d4858-2ad7-4f38-90c9-7070638157cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9" Oct 30 09:26:55.220492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.220385 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw94r\" (UniqueName: \"kubernetes.io/projected/053d4858-2ad7-4f38-90c9-7070638157cd-kube-api-access-lw94r\") pod \"maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9\" (UID: \"053d4858-2ad7-4f38-90c9-7070638157cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9" Oct 30 09:26:55.221093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.221070 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053d4858-2ad7-4f38-90c9-7070638157cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9\" (UID: \"053d4858-2ad7-4f38-90c9-7070638157cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9" Oct 30 09:26:55.223020 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.222999 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053d4858-2ad7-4f38-90c9-7070638157cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9\" (UID: \"053d4858-2ad7-4f38-90c9-7070638157cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9" Oct 30 09:26:55.269193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.269161 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw94r\" (UniqueName: \"kubernetes.io/projected/053d4858-2ad7-4f38-90c9-7070638157cd-kube-api-access-lw94r\") pod \"maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9\" (UID: \"053d4858-2ad7-4f38-90c9-7070638157cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9" Oct 30 09:26:55.355703 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.355601 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9" Oct 30 09:26:55.664445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.664415 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9"] Oct 30 09:26:55.666880 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:26:55.666848 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053d4858_2ad7_4f38_90c9_7070638157cd.slice/crio-887a205e3e6254d356a8269f8d02486aeea4d4f9deca0c22b9b2e72be08558dc WatchSource:0}: Error finding container 887a205e3e6254d356a8269f8d02486aeea4d4f9deca0c22b9b2e72be08558dc: Status 404 returned error can't find the container with id 887a205e3e6254d356a8269f8d02486aeea4d4f9deca0c22b9b2e72be08558dc Oct 30 09:26:55.863974 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:55.863934 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9" event={"ID":"053d4858-2ad7-4f38-90c9-7070638157cd","Type":"ContainerStarted","Data":"887a205e3e6254d356a8269f8d02486aeea4d4f9deca0c22b9b2e72be08558dc"} Oct 30 09:26:56.872047 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:56.871943 5938 generic.go:356] "Generic (PLEG): container finished" podID="053d4858-2ad7-4f38-90c9-7070638157cd" containerID="1fd26f6d84ef39a7afd55d671189cc20f5fe5be3b3b1b2e7e9dac5b56e2b167a" exitCode=0 Oct 30 09:26:56.872047 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:56.872034 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9" event={"ID":"053d4858-2ad7-4f38-90c9-7070638157cd","Type":"ContainerDied","Data":"1fd26f6d84ef39a7afd55d671189cc20f5fe5be3b3b1b2e7e9dac5b56e2b167a"} Oct 30 09:26:58.049382 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:58.049353 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9" Oct 30 09:26:58.161854 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:58.161753 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/053d4858-2ad7-4f38-90c9-7070638157cd-kube-api-access-lw94r\" (UniqueName: \"kubernetes.io/projected/053d4858-2ad7-4f38-90c9-7070638157cd-kube-api-access-lw94r\") pod \"053d4858-2ad7-4f38-90c9-7070638157cd\" (UID: \"053d4858-2ad7-4f38-90c9-7070638157cd\") " Oct 30 09:26:58.162030 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:58.161869 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/053d4858-2ad7-4f38-90c9-7070638157cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053d4858-2ad7-4f38-90c9-7070638157cd-maas-api-xks-ca\") pod \"053d4858-2ad7-4f38-90c9-7070638157cd\" (UID: \"053d4858-2ad7-4f38-90c9-7070638157cd\") " Oct 30 09:26:58.162030 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:58.161918 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/053d4858-2ad7-4f38-90c9-7070638157cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053d4858-2ad7-4f38-90c9-7070638157cd-maas-api-openshift-ca\") pod \"053d4858-2ad7-4f38-90c9-7070638157cd\" (UID: \"053d4858-2ad7-4f38-90c9-7070638157cd\") " Oct 30 09:26:58.162538 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:58.162505 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/053d4858-2ad7-4f38-90c9-7070638157cd-maas-api-openshift-ca" pod "053d4858-2ad7-4f38-90c9-7070638157cd" (UID: "053d4858-2ad7-4f38-90c9-7070638157cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:58.164372 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:58.164342 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/053d4858-2ad7-4f38-90c9-7070638157cd-maas-api-xks-ca" pod "053d4858-2ad7-4f38-90c9-7070638157cd" (UID: "053d4858-2ad7-4f38-90c9-7070638157cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:26:58.164539 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:58.164507 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053d4858-2ad7-4f38-90c9-7070638157cd-kube-api-access-lw94r" pod "053d4858-2ad7-4f38-90c9-7070638157cd" (UID: "053d4858-2ad7-4f38-90c9-7070638157cd"). InnerVolumeSpecName "kube-api-access-lw94r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:58.263468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:58.263430 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053d4858-2ad7-4f38-90c9-7070638157cd-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:58.263468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:58.263469 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053d4858-2ad7-4f38-90c9-7070638157cd-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:58.263468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:58.263480 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw94r\" (UniqueName: \"kubernetes.io/projected/053d4858-2ad7-4f38-90c9-7070638157cd-kube-api-access-lw94r\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:26:58.893630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:58.888740 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9" event={"ID":"053d4858-2ad7-4f38-90c9-7070638157cd","Type":"ContainerDied","Data":"887a205e3e6254d356a8269f8d02486aeea4d4f9deca0c22b9b2e72be08558dc"} Oct 30 09:26:58.893630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:58.888785 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9" Oct 30 09:26:58.893630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:26:58.888860 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="887a205e3e6254d356a8269f8d02486aeea4d4f9deca0c22b9b2e72be08558dc" Oct 30 09:27:15.814430 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:27:15.814390 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:27:18.497785 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.497745 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt"] Oct 30 09:27:18.498316 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.498302 5938 cpu_manager.go:414] "containerMap: removing container" podUID="053d4858-2ad7-4f38-90c9-7070638157cd" containerName="revoke-keys" Oct 30 09:27:18.498362 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.498318 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="053d4858-2ad7-4f38-90c9-7070638157cd" containerName="revoke-keys" Oct 30 09:27:18.498409 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.498400 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="053d4858-2ad7-4f38-90c9-7070638157cd" containerName="revoke-keys" Oct 30 09:27:18.498447 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.498413 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a20fda-9d11-47ca-887d-e1dac99e53a2" containerName="istio-proxy" Oct 30 09:27:18.502617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.502567 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" Oct 30 09:27:18.521105 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.521074 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt"] Oct 30 09:27:18.531918 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.531882 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-cbd96e\"" Oct 30 09:27:18.531918 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.531896 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-cbd96e-dockercfg-hqjgr\"" Oct 30 09:27:18.532570 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.532545 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwnc2\" (UniqueName: \"kubernetes.io/projected/93592883-03a8-4dfd-aefb-2ed6ae9ef6cc-kube-api-access-xwnc2\") pod \"payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt\" (UID: \"93592883-03a8-4dfd-aefb-2ed6ae9ef6cc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" Oct 30 09:27:18.532767 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.532736 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93592883-03a8-4dfd-aefb-2ed6ae9ef6cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt\" (UID: \"93592883-03a8-4dfd-aefb-2ed6ae9ef6cc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" Oct 30 09:27:18.648655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.648362 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwnc2\" (UniqueName: \"kubernetes.io/projected/93592883-03a8-4dfd-aefb-2ed6ae9ef6cc-kube-api-access-xwnc2\") pod \"payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt\" (UID: \"93592883-03a8-4dfd-aefb-2ed6ae9ef6cc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" Oct 30 09:27:18.648655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.648497 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93592883-03a8-4dfd-aefb-2ed6ae9ef6cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt\" (UID: \"93592883-03a8-4dfd-aefb-2ed6ae9ef6cc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" Oct 30 09:27:18.661613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.649546 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93592883-03a8-4dfd-aefb-2ed6ae9ef6cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt\" (UID: \"93592883-03a8-4dfd-aefb-2ed6ae9ef6cc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" Oct 30 09:27:18.685612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.680898 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22"] Oct 30 09:27:18.685612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.684891 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwnc2\" (UniqueName: \"kubernetes.io/projected/93592883-03a8-4dfd-aefb-2ed6ae9ef6cc-kube-api-access-xwnc2\") pod \"payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt\" (UID: \"93592883-03a8-4dfd-aefb-2ed6ae9ef6cc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" Oct 30 09:27:18.708543 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.695441 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22"] Oct 30 09:27:18.708543 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.695565 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" Oct 30 09:27:18.750146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.750035 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9018a44-392d-46e9-84a3-e62b2da99919-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22\" (UID: \"b9018a44-392d-46e9-84a3-e62b2da99919\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" Oct 30 09:27:18.750146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.750114 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2vhh\" (UniqueName: \"kubernetes.io/projected/b9018a44-392d-46e9-84a3-e62b2da99919-kube-api-access-d2vhh\") pod \"payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22\" (UID: \"b9018a44-392d-46e9-84a3-e62b2da99919\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" Oct 30 09:27:18.817512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.817475 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" Oct 30 09:27:18.851451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.851405 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9018a44-392d-46e9-84a3-e62b2da99919-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22\" (UID: \"b9018a44-392d-46e9-84a3-e62b2da99919\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" Oct 30 09:27:18.851688 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.851460 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vhh\" (UniqueName: \"kubernetes.io/projected/b9018a44-392d-46e9-84a3-e62b2da99919-kube-api-access-d2vhh\") pod \"payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22\" (UID: \"b9018a44-392d-46e9-84a3-e62b2da99919\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" Oct 30 09:27:18.852295 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.852274 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9018a44-392d-46e9-84a3-e62b2da99919-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22\" (UID: \"b9018a44-392d-46e9-84a3-e62b2da99919\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" Oct 30 09:27:18.873323 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:18.873293 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2vhh\" (UniqueName: \"kubernetes.io/projected/b9018a44-392d-46e9-84a3-e62b2da99919-kube-api-access-d2vhh\") pod \"payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22\" (UID: \"b9018a44-392d-46e9-84a3-e62b2da99919\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" Oct 30 09:27:19.022900 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:19.022801 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" Oct 30 09:27:19.125718 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:19.125066 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt"] Oct 30 09:27:19.128092 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:27:19.128046 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93592883_03a8_4dfd_aefb_2ed6ae9ef6cc.slice/crio-a9a7b2412ee46185a225bda7ec51e101ba584e0286606ca3651e6c4cdf34bd43 WatchSource:0}: Error finding container a9a7b2412ee46185a225bda7ec51e101ba584e0286606ca3651e6c4cdf34bd43: Status 404 returned error can't find the container with id a9a7b2412ee46185a225bda7ec51e101ba584e0286606ca3651e6c4cdf34bd43 Oct 30 09:27:19.328741 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:19.328710 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22"] Oct 30 09:27:19.330294 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:27:19.330265 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9018a44_392d_46e9_84a3_e62b2da99919.slice/crio-81874bff1a4899000dcd93e91e3e8a8cc1571e4b95a81d1a9dd2bc20c5d51573 WatchSource:0}: Error finding container 81874bff1a4899000dcd93e91e3e8a8cc1571e4b95a81d1a9dd2bc20c5d51573: Status 404 returned error can't find the container with id 81874bff1a4899000dcd93e91e3e8a8cc1571e4b95a81d1a9dd2bc20c5d51573 Oct 30 09:27:20.025154 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:20.025115 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" event={"ID":"b9018a44-392d-46e9-84a3-e62b2da99919","Type":"ContainerStarted","Data":"5527334ae9891b0e10b74e0ee76770968ab407e49bf71fe5c166464a397d4c11"} Oct 30 09:27:20.028978 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:20.028799 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" event={"ID":"b9018a44-392d-46e9-84a3-e62b2da99919","Type":"ContainerStarted","Data":"81874bff1a4899000dcd93e91e3e8a8cc1571e4b95a81d1a9dd2bc20c5d51573"} Oct 30 09:27:20.028978 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:20.028907 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" Oct 30 09:27:20.032383 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:20.032272 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" event={"ID":"93592883-03a8-4dfd-aefb-2ed6ae9ef6cc","Type":"ContainerStarted","Data":"da5067eadb445687a11e61e92b738ef464a643b28ce296333f6935053971d305"} Oct 30 09:27:20.032383 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:20.032340 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" event={"ID":"93592883-03a8-4dfd-aefb-2ed6ae9ef6cc","Type":"ContainerStarted","Data":"a9a7b2412ee46185a225bda7ec51e101ba584e0286606ca3651e6c4cdf34bd43"} Oct 30 09:27:20.032963 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:20.032476 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" Oct 30 09:27:20.069370 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:20.069314 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" podStartSLOduration=2.069297844 podStartE2EDuration="2.069297844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:20.067778044 +0000 UTC m=+1957.930483866" watchObservedRunningTime="2026-10-30 09:27:20.069297844 +0000 UTC m=+1957.932003688" Oct 30 09:27:20.111426 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:20.111377 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" podStartSLOduration=2.111363307 podStartE2EDuration="2.111363307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:20.110675861 +0000 UTC m=+1957.973381685" watchObservedRunningTime="2026-10-30 09:27:20.111363307 +0000 UTC m=+1957.974069122" Oct 30 09:27:25.245058 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.245023 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l"] Oct 30 09:27:25.249478 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.249449 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:27:25.268629 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.268556 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l"] Oct 30 09:27:25.278797 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.278758 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-d3de9d-serving-cert\"" Oct 30 09:27:25.348256 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.348218 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/461caf97-a7d2-47ad-ba94-252492172b2d-metrics-certs\") pod \"maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l\" (UID: \"461caf97-a7d2-47ad-ba94-252492172b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:27:25.348473 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.348276 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/461caf97-a7d2-47ad-ba94-252492172b2d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l\" (UID: \"461caf97-a7d2-47ad-ba94-252492172b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:27:25.348473 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.348437 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/461caf97-a7d2-47ad-ba94-252492172b2d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l\" (UID: \"461caf97-a7d2-47ad-ba94-252492172b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:27:25.348630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.348492 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn8js\" (UniqueName: \"kubernetes.io/projected/461caf97-a7d2-47ad-ba94-252492172b2d-kube-api-access-dn8js\") pod \"maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l\" (UID: \"461caf97-a7d2-47ad-ba94-252492172b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:27:25.449993 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.449954 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/461caf97-a7d2-47ad-ba94-252492172b2d-metrics-certs\") pod \"maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l\" (UID: \"461caf97-a7d2-47ad-ba94-252492172b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:27:25.450306 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.450284 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/461caf97-a7d2-47ad-ba94-252492172b2d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l\" (UID: \"461caf97-a7d2-47ad-ba94-252492172b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:27:25.450481 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.450463 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/461caf97-a7d2-47ad-ba94-252492172b2d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l\" (UID: \"461caf97-a7d2-47ad-ba94-252492172b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:27:25.450765 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.450741 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn8js\" (UniqueName: \"kubernetes.io/projected/461caf97-a7d2-47ad-ba94-252492172b2d-kube-api-access-dn8js\") pod \"maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l\" (UID: \"461caf97-a7d2-47ad-ba94-252492172b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:27:25.451301 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.451271 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/461caf97-a7d2-47ad-ba94-252492172b2d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l\" (UID: \"461caf97-a7d2-47ad-ba94-252492172b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:27:25.455856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.455825 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/461caf97-a7d2-47ad-ba94-252492172b2d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l\" (UID: \"461caf97-a7d2-47ad-ba94-252492172b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:27:25.457156 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.457126 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/461caf97-a7d2-47ad-ba94-252492172b2d-metrics-certs\") pod \"maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l\" (UID: \"461caf97-a7d2-47ad-ba94-252492172b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:27:25.474206 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.474170 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn8js\" (UniqueName: \"kubernetes.io/projected/461caf97-a7d2-47ad-ba94-252492172b2d-kube-api-access-dn8js\") pod \"maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l\" (UID: \"461caf97-a7d2-47ad-ba94-252492172b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:27:25.564684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.564534 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:27:25.872375 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.872342 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l"] Oct 30 09:27:25.873905 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:27:25.873880 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461caf97_a7d2_47ad_ba94_252492172b2d.slice/crio-32d15464c1319b14d3c6960f426202bb568246fa4bd9140a8cd6390ec270ee68 WatchSource:0}: Error finding container 32d15464c1319b14d3c6960f426202bb568246fa4bd9140a8cd6390ec270ee68: Status 404 returned error can't find the container with id 32d15464c1319b14d3c6960f426202bb568246fa4bd9140a8cd6390ec270ee68 Oct 30 09:27:25.875368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:25.875346 5938 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:27:26.079868 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:26.079831 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" event={"ID":"461caf97-a7d2-47ad-ba94-252492172b2d","Type":"ContainerStarted","Data":"32d15464c1319b14d3c6960f426202bb568246fa4bd9140a8cd6390ec270ee68"} Oct 30 09:27:27.100735 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:27.100696 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" event={"ID":"461caf97-a7d2-47ad-ba94-252492172b2d","Type":"ContainerStarted","Data":"d6898d8bcfbe8b438baf34477d21ff92217679dc39124982e9c77d0599595370"} Oct 30 09:27:27.101302 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:27.100911 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:27:27.148482 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:27.148420 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" podStartSLOduration=1.790892205 podStartE2EDuration="2.148402732s" totalImagesPullingTime="357.510527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:27.145867711 +0000 UTC m=+1965.008573533" watchObservedRunningTime="2026-10-30 09:27:27.148402732 +0000 UTC m=+1965.011108566" Oct 30 09:27:31.042080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:31.042047 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" Oct 30 09:27:31.042684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:31.042137 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" Oct 30 09:27:33.112956 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:33.112927 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:27:45.812331 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:27:45.812271 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:27:46.660574 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.660541 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9"] Oct 30 09:27:46.664505 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.664484 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.687503 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.687448 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9"] Oct 30 09:27:46.764118 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.764082 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-workload-certs\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.764322 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.764141 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-workload-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.764322 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.764228 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-data\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.764322 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.764306 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-token\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.764430 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.764396 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993fc75c-be0b-49a1-8298-a3dd75e67186-istiod-ca-cert\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.764489 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.764473 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-credential-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.764524 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.764497 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-envoy\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.764524 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.764515 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pbhn\" (UniqueName: \"kubernetes.io/projected/993fc75c-be0b-49a1-8298-a3dd75e67186-kube-api-access-2pbhn\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.764635 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.764577 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-podinfo\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.770569 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.770547 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-d3de9d-openshift-default-dockercfg-6cq7q\"" Oct 30 09:27:46.865485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.865447 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-workload-certs\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.865485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.865500 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-workload-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.866086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.865537 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-data\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.866086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.865647 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-token\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.866086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.865684 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993fc75c-be0b-49a1-8298-a3dd75e67186-istiod-ca-cert\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.866086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.865731 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-credential-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.866086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.865777 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-envoy\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.866086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.865803 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pbhn\" (UniqueName: \"kubernetes.io/projected/993fc75c-be0b-49a1-8298-a3dd75e67186-kube-api-access-2pbhn\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.866086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.865890 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-podinfo\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.866086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.865964 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-workload-certs\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.866086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.866033 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-data\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.866462 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.866294 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-workload-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.866462 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.866334 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-credential-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.866913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.866889 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993fc75c-be0b-49a1-8298-a3dd75e67186-istiod-ca-cert\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.868621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.868560 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-envoy\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.868770 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.868751 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-podinfo\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.886864 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.886830 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-token\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.887025 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.886970 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pbhn\" (UniqueName: \"kubernetes.io/projected/993fc75c-be0b-49a1-8298-a3dd75e67186-kube-api-access-2pbhn\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:46.982422 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:46.982320 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:47.319672 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:47.318879 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9"] Oct 30 09:27:47.344856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:47.344169 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:27:47.344856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:47.344271 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:27:47.344856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:47.344322 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:27:47.877730 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:47.877696 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9"] Oct 30 09:27:47.882103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:47.882064 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5491bf-7c8p9"] Oct 30 09:27:48.257894 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:48.257812 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="053d4858-2ad7-4f38-90c9-7070638157cd" path="/var/lib/kubelet/pods/053d4858-2ad7-4f38-90c9-7070638157cd/volumes" Oct 30 09:27:48.258126 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:48.258108 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" event={"ID":"993fc75c-be0b-49a1-8298-a3dd75e67186","Type":"ContainerStarted","Data":"afa0ec2ebc8e85c41b953c110dac649af6b24c2c278f46d84a1333f2f40be5a6"} Oct 30 09:27:48.258170 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:48.258156 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" event={"ID":"993fc75c-be0b-49a1-8298-a3dd75e67186","Type":"ContainerStarted","Data":"eaf7ea0fc7a8e0cb4d30cfbbc61e2aa9d11c8c3b90ae2f9a51cc658688b3dd52"} Oct 30 09:27:48.299443 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:48.299389 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" podStartSLOduration=2.299374324 podStartE2EDuration="2.299374324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:48.297191671 +0000 UTC m=+1986.159897494" watchObservedRunningTime="2026-10-30 09:27:48.299374324 +0000 UTC m=+1986.162080144" Oct 30 09:27:48.983275 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:48.983235 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:48.984863 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:48.984828 5938 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 09:27:48.984965 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:48.984894 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" podUID="993fc75c-be0b-49a1-8298-a3dd75e67186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 09:27:49.985616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:49.984025 5938 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 09:27:49.985616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:49.984086 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" podUID="993fc75c-be0b-49a1-8298-a3dd75e67186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 09:27:50.983258 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:50.983206 5938 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 09:27:50.983491 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:50.983293 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" podUID="993fc75c-be0b-49a1-8298-a3dd75e67186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 09:27:51.983642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:51.983576 5938 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 09:27:51.984065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:51.983658 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" podUID="993fc75c-be0b-49a1-8298-a3dd75e67186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 09:27:52.983753 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:52.983715 5938 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 09:27:52.984163 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:52.983775 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" podUID="993fc75c-be0b-49a1-8298-a3dd75e67186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 09:27:54.222879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:54.222842 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:54.223318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:54.223192 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:54.224315 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:54.224294 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:27:58.585229 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:58.585179 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf"] Oct 30 09:27:58.594291 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:58.594253 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf" Oct 30 09:27:58.606090 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:58.606051 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf"] Oct 30 09:27:58.618494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:58.618461 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:27:58.620157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:58.620131 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:27:58.699831 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:58.699794 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf\" (UID: \"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf" Oct 30 09:27:58.700033 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:58.699893 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf\" (UID: \"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf" Oct 30 09:27:58.700033 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:58.699943 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfrqk\" (UniqueName: \"kubernetes.io/projected/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-kube-api-access-jfrqk\") pod \"maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf\" (UID: \"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf" Oct 30 09:27:58.800956 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:58.800913 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf\" (UID: \"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf" Oct 30 09:27:58.801176 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:58.801014 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfrqk\" (UniqueName: \"kubernetes.io/projected/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-kube-api-access-jfrqk\") pod \"maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf\" (UID: \"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf" Oct 30 09:27:58.801263 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:58.801212 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf\" (UID: \"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf" Oct 30 09:27:58.801791 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:58.801766 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf\" (UID: \"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf" Oct 30 09:27:58.803976 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:58.803947 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf\" (UID: \"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf" Oct 30 09:27:58.821429 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:58.821397 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfrqk\" (UniqueName: \"kubernetes.io/projected/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-kube-api-access-jfrqk\") pod \"maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf\" (UID: \"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf" Oct 30 09:27:58.919814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:58.919770 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf" Oct 30 09:27:59.228202 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:59.228164 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf"] Oct 30 09:27:59.229925 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:27:59.229894 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524777b6_21b1_42cb_a2c9_0a69ec4bc0bf.slice/crio-2f6fe2e4410bd470ada54e0dcc674e12fb8dbad2ff40a5d0d04720ca3a7a8bc8 WatchSource:0}: Error finding container 2f6fe2e4410bd470ada54e0dcc674e12fb8dbad2ff40a5d0d04720ca3a7a8bc8: Status 404 returned error can't find the container with id 2f6fe2e4410bd470ada54e0dcc674e12fb8dbad2ff40a5d0d04720ca3a7a8bc8 Oct 30 09:27:59.364226 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:27:59.364190 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf" event={"ID":"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf","Type":"ContainerStarted","Data":"2f6fe2e4410bd470ada54e0dcc674e12fb8dbad2ff40a5d0d04720ca3a7a8bc8"} Oct 30 09:28:00.372244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:00.372208 5938 generic.go:356] "Generic (PLEG): container finished" podID="524777b6-21b1-42cb-a2c9-0a69ec4bc0bf" containerID="28e9e33ea7238b9b78196c0403903e5014b91792dc65e79c17b330e3ebf01007" exitCode=0 Oct 30 09:28:00.372750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:00.372253 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf" event={"ID":"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf","Type":"ContainerDied","Data":"28e9e33ea7238b9b78196c0403903e5014b91792dc65e79c17b330e3ebf01007"} Oct 30 09:28:01.543556 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:01.543520 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf" Oct 30 09:28:01.652220 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:01.652128 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-maas-api-xks-ca\") pod \"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf\" (UID: \"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf\") " Oct 30 09:28:01.652220 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:01.652178 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-maas-api-openshift-ca\") pod \"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf\" (UID: \"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf\") " Oct 30 09:28:01.652425 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:01.652271 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-kube-api-access-jfrqk\" (UniqueName: \"kubernetes.io/projected/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-kube-api-access-jfrqk\") pod \"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf\" (UID: \"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf\") " Oct 30 09:28:01.652647 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:01.652612 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-maas-api-openshift-ca" pod "524777b6-21b1-42cb-a2c9-0a69ec4bc0bf" (UID: "524777b6-21b1-42cb-a2c9-0a69ec4bc0bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:01.652975 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:01.652960 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:28:01.654488 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:01.654465 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-maas-api-xks-ca" pod "524777b6-21b1-42cb-a2c9-0a69ec4bc0bf" (UID: "524777b6-21b1-42cb-a2c9-0a69ec4bc0bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:28:01.654603 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:01.654569 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-kube-api-access-jfrqk" pod "524777b6-21b1-42cb-a2c9-0a69ec4bc0bf" (UID: "524777b6-21b1-42cb-a2c9-0a69ec4bc0bf"). InnerVolumeSpecName "kube-api-access-jfrqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:01.754624 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:01.754552 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:28:01.754624 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:01.754602 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfrqk\" (UniqueName: \"kubernetes.io/projected/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf-kube-api-access-jfrqk\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:28:02.388657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:02.388614 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf" event={"ID":"524777b6-21b1-42cb-a2c9-0a69ec4bc0bf","Type":"ContainerDied","Data":"2f6fe2e4410bd470ada54e0dcc674e12fb8dbad2ff40a5d0d04720ca3a7a8bc8"} Oct 30 09:28:02.388842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:02.388688 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f6fe2e4410bd470ada54e0dcc674e12fb8dbad2ff40a5d0d04720ca3a7a8bc8" Oct 30 09:28:02.388842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:02.388691 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf" Oct 30 09:28:08.395445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.395404 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w"] Oct 30 09:28:08.396093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.396074 5938 cpu_manager.go:414] "containerMap: removing container" podUID="524777b6-21b1-42cb-a2c9-0a69ec4bc0bf" containerName="revoke-keys" Oct 30 09:28:08.396140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.396095 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="524777b6-21b1-42cb-a2c9-0a69ec4bc0bf" containerName="revoke-keys" Oct 30 09:28:08.396220 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.396210 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="524777b6-21b1-42cb-a2c9-0a69ec4bc0bf" containerName="revoke-keys" Oct 30 09:28:08.399780 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.399759 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" Oct 30 09:28:08.419082 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.419041 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w"] Oct 30 09:28:08.432638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.429646 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:28:08.432638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.429709 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:28:08.575563 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.575122 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca651833-5ed5-4134-92b5-00a99c319e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w\" (UID: \"ca651833-5ed5-4134-92b5-00a99c319e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" Oct 30 09:28:08.575963 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.575912 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g87w7\" (UniqueName: \"kubernetes.io/projected/ca651833-5ed5-4134-92b5-00a99c319e2e-kube-api-access-g87w7\") pod \"maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w\" (UID: \"ca651833-5ed5-4134-92b5-00a99c319e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" Oct 30 09:28:08.576259 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.576222 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca651833-5ed5-4134-92b5-00a99c319e2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w\" (UID: \"ca651833-5ed5-4134-92b5-00a99c319e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" Oct 30 09:28:08.678380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.678278 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca651833-5ed5-4134-92b5-00a99c319e2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w\" (UID: \"ca651833-5ed5-4134-92b5-00a99c319e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" Oct 30 09:28:08.678547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.678494 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca651833-5ed5-4134-92b5-00a99c319e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w\" (UID: \"ca651833-5ed5-4134-92b5-00a99c319e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" Oct 30 09:28:08.678547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.678528 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g87w7\" (UniqueName: \"kubernetes.io/projected/ca651833-5ed5-4134-92b5-00a99c319e2e-kube-api-access-g87w7\") pod \"maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w\" (UID: \"ca651833-5ed5-4134-92b5-00a99c319e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" Oct 30 09:28:08.679290 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.679218 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca651833-5ed5-4134-92b5-00a99c319e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w\" (UID: \"ca651833-5ed5-4134-92b5-00a99c319e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" Oct 30 09:28:08.680977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.680951 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca651833-5ed5-4134-92b5-00a99c319e2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w\" (UID: \"ca651833-5ed5-4134-92b5-00a99c319e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" Oct 30 09:28:08.700221 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.700189 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g87w7\" (UniqueName: \"kubernetes.io/projected/ca651833-5ed5-4134-92b5-00a99c319e2e-kube-api-access-g87w7\") pod \"maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w\" (UID: \"ca651833-5ed5-4134-92b5-00a99c319e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" Oct 30 09:28:08.716077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:08.716036 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" Oct 30 09:28:09.021152 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:09.021104 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w"] Oct 30 09:28:09.022561 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:28:09.022534 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca651833_5ed5_4134_92b5_00a99c319e2e.slice/crio-cfd719a9ab0d0dff8b650bc5f5b43ba58113f71a88b495b3ca608eb39357e30a WatchSource:0}: Error finding container cfd719a9ab0d0dff8b650bc5f5b43ba58113f71a88b495b3ca608eb39357e30a: Status 404 returned error can't find the container with id cfd719a9ab0d0dff8b650bc5f5b43ba58113f71a88b495b3ca608eb39357e30a Oct 30 09:28:09.585628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:09.584462 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" event={"ID":"ca651833-5ed5-4134-92b5-00a99c319e2e","Type":"ContainerStarted","Data":"6ae07e9659cf64e83e872578ae582f2f6589b748bad5ed6a13f2f06d686309fc"} Oct 30 09:28:09.585628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:09.584531 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" event={"ID":"ca651833-5ed5-4134-92b5-00a99c319e2e","Type":"ContainerStarted","Data":"cfd719a9ab0d0dff8b650bc5f5b43ba58113f71a88b495b3ca608eb39357e30a"} Oct 30 09:28:09.629244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:09.629177 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" podStartSLOduration=1.629156584 podStartE2EDuration="1.629156584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:28:09.628039493 +0000 UTC m=+2007.490745318" watchObservedRunningTime="2026-10-30 09:28:09.629156584 +0000 UTC m=+2007.491862422" Oct 30 09:28:10.592981 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:10.592945 5938 generic.go:356] "Generic (PLEG): container finished" podID="ca651833-5ed5-4134-92b5-00a99c319e2e" containerID="6ae07e9659cf64e83e872578ae582f2f6589b748bad5ed6a13f2f06d686309fc" exitCode=0 Oct 30 09:28:10.593437 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:10.593019 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" event={"ID":"ca651833-5ed5-4134-92b5-00a99c319e2e","Type":"ContainerDied","Data":"6ae07e9659cf64e83e872578ae582f2f6589b748bad5ed6a13f2f06d686309fc"} Oct 30 09:28:11.766977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:11.766947 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" Oct 30 09:28:11.927932 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:11.927891 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca651833-5ed5-4134-92b5-00a99c319e2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca651833-5ed5-4134-92b5-00a99c319e2e-maas-api-xks-ca\") pod \"ca651833-5ed5-4134-92b5-00a99c319e2e\" (UID: \"ca651833-5ed5-4134-92b5-00a99c319e2e\") " Oct 30 09:28:11.928110 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:11.927952 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca651833-5ed5-4134-92b5-00a99c319e2e-kube-api-access-g87w7\" (UniqueName: \"kubernetes.io/projected/ca651833-5ed5-4134-92b5-00a99c319e2e-kube-api-access-g87w7\") pod \"ca651833-5ed5-4134-92b5-00a99c319e2e\" (UID: \"ca651833-5ed5-4134-92b5-00a99c319e2e\") " Oct 30 09:28:11.928110 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:11.928058 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca651833-5ed5-4134-92b5-00a99c319e2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca651833-5ed5-4134-92b5-00a99c319e2e-maas-api-openshift-ca\") pod \"ca651833-5ed5-4134-92b5-00a99c319e2e\" (UID: \"ca651833-5ed5-4134-92b5-00a99c319e2e\") " Oct 30 09:28:11.928428 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:11.928400 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca651833-5ed5-4134-92b5-00a99c319e2e-maas-api-openshift-ca" pod "ca651833-5ed5-4134-92b5-00a99c319e2e" (UID: "ca651833-5ed5-4134-92b5-00a99c319e2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:11.928679 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:11.928659 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca651833-5ed5-4134-92b5-00a99c319e2e-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:28:11.930301 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:11.930279 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca651833-5ed5-4134-92b5-00a99c319e2e-maas-api-xks-ca" pod "ca651833-5ed5-4134-92b5-00a99c319e2e" (UID: "ca651833-5ed5-4134-92b5-00a99c319e2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:28:11.930377 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:11.930308 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca651833-5ed5-4134-92b5-00a99c319e2e-kube-api-access-g87w7" pod "ca651833-5ed5-4134-92b5-00a99c319e2e" (UID: "ca651833-5ed5-4134-92b5-00a99c319e2e"). InnerVolumeSpecName "kube-api-access-g87w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:12.029924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:12.029886 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca651833-5ed5-4134-92b5-00a99c319e2e-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:28:12.029924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:12.029928 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g87w7\" (UniqueName: \"kubernetes.io/projected/ca651833-5ed5-4134-92b5-00a99c319e2e-kube-api-access-g87w7\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:28:12.608177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:12.608097 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" Oct 30 09:28:12.608332 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:12.608096 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w" event={"ID":"ca651833-5ed5-4134-92b5-00a99c319e2e","Type":"ContainerDied","Data":"cfd719a9ab0d0dff8b650bc5f5b43ba58113f71a88b495b3ca608eb39357e30a"} Oct 30 09:28:12.608332 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:12.608220 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfd719a9ab0d0dff8b650bc5f5b43ba58113f71a88b495b3ca608eb39357e30a" Oct 30 09:28:15.817025 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:28:15.816976 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:28:32.608670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:32.608627 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf"] Oct 30 09:28:32.621822 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:32.621785 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-19a563-2pnxf"] Oct 30 09:28:34.259575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:34.259534 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="524777b6-21b1-42cb-a2c9-0a69ec4bc0bf" path="/var/lib/kubelet/pods/524777b6-21b1-42cb-a2c9-0a69ec4bc0bf/volumes" Oct 30 09:28:40.258266 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.258228 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc"] Oct 30 09:28:40.258846 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.258829 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ca651833-5ed5-4134-92b5-00a99c319e2e" containerName="revoke-keys" Oct 30 09:28:40.258900 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.258847 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca651833-5ed5-4134-92b5-00a99c319e2e" containerName="revoke-keys" Oct 30 09:28:40.258946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.258936 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca651833-5ed5-4134-92b5-00a99c319e2e" containerName="revoke-keys" Oct 30 09:28:40.262803 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.262779 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" Oct 30 09:28:40.281521 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.281483 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc"] Oct 30 09:28:40.291770 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.291735 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:28:40.292011 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.291822 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:28:40.299350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.299293 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e571e8-52e5-4916-9c48-e83c8279edac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc\" (UID: \"29e571e8-52e5-4916-9c48-e83c8279edac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" Oct 30 09:28:40.299551 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.299497 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e571e8-52e5-4916-9c48-e83c8279edac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc\" (UID: \"29e571e8-52e5-4916-9c48-e83c8279edac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" Oct 30 09:28:40.299672 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.299558 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8ftj\" (UniqueName: \"kubernetes.io/projected/29e571e8-52e5-4916-9c48-e83c8279edac-kube-api-access-m8ftj\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc\" (UID: \"29e571e8-52e5-4916-9c48-e83c8279edac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" Oct 30 09:28:40.401517 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.401452 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e571e8-52e5-4916-9c48-e83c8279edac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc\" (UID: \"29e571e8-52e5-4916-9c48-e83c8279edac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" Oct 30 09:28:40.401857 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.401655 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e571e8-52e5-4916-9c48-e83c8279edac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc\" (UID: \"29e571e8-52e5-4916-9c48-e83c8279edac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" Oct 30 09:28:40.401857 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.401704 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ftj\" (UniqueName: \"kubernetes.io/projected/29e571e8-52e5-4916-9c48-e83c8279edac-kube-api-access-m8ftj\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc\" (UID: \"29e571e8-52e5-4916-9c48-e83c8279edac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" Oct 30 09:28:40.402763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.402721 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e571e8-52e5-4916-9c48-e83c8279edac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc\" (UID: \"29e571e8-52e5-4916-9c48-e83c8279edac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" Oct 30 09:28:40.405749 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.405695 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e571e8-52e5-4916-9c48-e83c8279edac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc\" (UID: \"29e571e8-52e5-4916-9c48-e83c8279edac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" Oct 30 09:28:40.422720 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.422691 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ftj\" (UniqueName: \"kubernetes.io/projected/29e571e8-52e5-4916-9c48-e83c8279edac-kube-api-access-m8ftj\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc\" (UID: \"29e571e8-52e5-4916-9c48-e83c8279edac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" Oct 30 09:28:40.577665 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.577528 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" Oct 30 09:28:40.889282 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:40.889171 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc"] Oct 30 09:28:40.892015 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:28:40.891972 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e571e8_52e5_4916_9c48_e83c8279edac.slice/crio-0a230e1dbb975726b1c48498a20ef51a466a348a99f404a596732d8fa7c760b3 WatchSource:0}: Error finding container 0a230e1dbb975726b1c48498a20ef51a466a348a99f404a596732d8fa7c760b3: Status 404 returned error can't find the container with id 0a230e1dbb975726b1c48498a20ef51a466a348a99f404a596732d8fa7c760b3 Oct 30 09:28:41.854299 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:41.854253 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" event={"ID":"29e571e8-52e5-4916-9c48-e83c8279edac","Type":"ContainerStarted","Data":"cd568b90c4efc1ced627c3429f39505a5626a96047250d7ce6d9e98b58bfab9f"} Oct 30 09:28:41.854820 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:41.854311 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" event={"ID":"29e571e8-52e5-4916-9c48-e83c8279edac","Type":"ContainerStarted","Data":"0a230e1dbb975726b1c48498a20ef51a466a348a99f404a596732d8fa7c760b3"} Oct 30 09:28:41.895730 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:41.895677 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" podStartSLOduration=1.895662559 podStartE2EDuration="1.895662559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:28:41.895326101 +0000 UTC m=+2039.758031924" watchObservedRunningTime="2026-10-30 09:28:41.895662559 +0000 UTC m=+2039.758368381" Oct 30 09:28:42.861580 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:42.861539 5938 generic.go:356] "Generic (PLEG): container finished" podID="29e571e8-52e5-4916-9c48-e83c8279edac" containerID="cd568b90c4efc1ced627c3429f39505a5626a96047250d7ce6d9e98b58bfab9f" exitCode=0 Oct 30 09:28:42.862139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:42.861575 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" event={"ID":"29e571e8-52e5-4916-9c48-e83c8279edac","Type":"ContainerDied","Data":"cd568b90c4efc1ced627c3429f39505a5626a96047250d7ce6d9e98b58bfab9f"} Oct 30 09:28:44.173142 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:44.173106 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" Oct 30 09:28:44.277492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:44.277459 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e571e8-52e5-4916-9c48-e83c8279edac-kube-api-access-m8ftj\" (UniqueName: \"kubernetes.io/projected/29e571e8-52e5-4916-9c48-e83c8279edac-kube-api-access-m8ftj\") pod \"29e571e8-52e5-4916-9c48-e83c8279edac\" (UID: \"29e571e8-52e5-4916-9c48-e83c8279edac\") " Oct 30 09:28:44.277715 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:44.277499 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29e571e8-52e5-4916-9c48-e83c8279edac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e571e8-52e5-4916-9c48-e83c8279edac-maas-api-openshift-ca\") pod \"29e571e8-52e5-4916-9c48-e83c8279edac\" (UID: \"29e571e8-52e5-4916-9c48-e83c8279edac\") " Oct 30 09:28:44.277715 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:44.277699 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29e571e8-52e5-4916-9c48-e83c8279edac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e571e8-52e5-4916-9c48-e83c8279edac-maas-api-xks-ca\") pod \"29e571e8-52e5-4916-9c48-e83c8279edac\" (UID: \"29e571e8-52e5-4916-9c48-e83c8279edac\") " Oct 30 09:28:44.277956 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:44.277924 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e571e8-52e5-4916-9c48-e83c8279edac-maas-api-openshift-ca" pod "29e571e8-52e5-4916-9c48-e83c8279edac" (UID: "29e571e8-52e5-4916-9c48-e83c8279edac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:44.278396 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:44.278349 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e571e8-52e5-4916-9c48-e83c8279edac-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:28:44.280028 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:44.279997 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e571e8-52e5-4916-9c48-e83c8279edac-maas-api-xks-ca" pod "29e571e8-52e5-4916-9c48-e83c8279edac" (UID: "29e571e8-52e5-4916-9c48-e83c8279edac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:28:44.280028 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:44.280013 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e571e8-52e5-4916-9c48-e83c8279edac-kube-api-access-m8ftj" pod "29e571e8-52e5-4916-9c48-e83c8279edac" (UID: "29e571e8-52e5-4916-9c48-e83c8279edac"). InnerVolumeSpecName "kube-api-access-m8ftj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:44.379491 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:44.379450 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8ftj\" (UniqueName: \"kubernetes.io/projected/29e571e8-52e5-4916-9c48-e83c8279edac-kube-api-access-m8ftj\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:28:44.379491 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:44.379488 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e571e8-52e5-4916-9c48-e83c8279edac-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:28:44.445966 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:44.445847 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w"] Oct 30 09:28:44.448673 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:44.448641 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-20f71c-s8g2w"] Oct 30 09:28:44.880789 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:44.880754 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" Oct 30 09:28:44.880973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:44.880743 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc" event={"ID":"29e571e8-52e5-4916-9c48-e83c8279edac","Type":"ContainerDied","Data":"0a230e1dbb975726b1c48498a20ef51a466a348a99f404a596732d8fa7c760b3"} Oct 30 09:28:44.880973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:44.880878 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a230e1dbb975726b1c48498a20ef51a466a348a99f404a596732d8fa7c760b3" Oct 30 09:28:45.810961 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:28:45.810925 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:28:46.257905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:28:46.257868 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca651833-5ed5-4134-92b5-00a99c319e2e" path="/var/lib/kubelet/pods/ca651833-5ed5-4134-92b5-00a99c319e2e/volumes" Oct 30 09:29:02.965922 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:02.965882 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22"] Oct 30 09:29:03.001120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:03.001086 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt"] Oct 30 09:29:06.181910 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.181868 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l"] Oct 30 09:29:06.182439 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.182155 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" podUID="461caf97-a7d2-47ad-ba94-252492172b2d" containerName="maas-api" containerID="cri-o://d6898d8bcfbe8b438baf34477d21ff92217679dc39124982e9c77d0599595370" gracePeriod=30 Oct 30 09:29:06.630292 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.629945 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:29:06.731634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.730153 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/461caf97-a7d2-47ad-ba94-252492172b2d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/461caf97-a7d2-47ad-ba94-252492172b2d-metrics-certs\") pod \"461caf97-a7d2-47ad-ba94-252492172b2d\" (UID: \"461caf97-a7d2-47ad-ba94-252492172b2d\") " Oct 30 09:29:06.731634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.730302 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/461caf97-a7d2-47ad-ba94-252492172b2d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/461caf97-a7d2-47ad-ba94-252492172b2d-openshift-service-ca\") pod \"461caf97-a7d2-47ad-ba94-252492172b2d\" (UID: \"461caf97-a7d2-47ad-ba94-252492172b2d\") " Oct 30 09:29:06.731634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.730352 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/461caf97-a7d2-47ad-ba94-252492172b2d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/461caf97-a7d2-47ad-ba94-252492172b2d-maas-api-tls\") pod \"461caf97-a7d2-47ad-ba94-252492172b2d\" (UID: \"461caf97-a7d2-47ad-ba94-252492172b2d\") " Oct 30 09:29:06.731634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.730410 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/461caf97-a7d2-47ad-ba94-252492172b2d-kube-api-access-dn8js\" (UniqueName: \"kubernetes.io/projected/461caf97-a7d2-47ad-ba94-252492172b2d-kube-api-access-dn8js\") pod \"461caf97-a7d2-47ad-ba94-252492172b2d\" (UID: \"461caf97-a7d2-47ad-ba94-252492172b2d\") " Oct 30 09:29:06.732186 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.731725 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/461caf97-a7d2-47ad-ba94-252492172b2d-openshift-service-ca" pod "461caf97-a7d2-47ad-ba94-252492172b2d" (UID: "461caf97-a7d2-47ad-ba94-252492172b2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:06.736184 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.736123 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461caf97-a7d2-47ad-ba94-252492172b2d-kube-api-access-dn8js" pod "461caf97-a7d2-47ad-ba94-252492172b2d" (UID: "461caf97-a7d2-47ad-ba94-252492172b2d"). InnerVolumeSpecName "kube-api-access-dn8js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:06.751798 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.751743 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/461caf97-a7d2-47ad-ba94-252492172b2d-maas-api-tls" pod "461caf97-a7d2-47ad-ba94-252492172b2d" (UID: "461caf97-a7d2-47ad-ba94-252492172b2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:06.752008 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.751847 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/461caf97-a7d2-47ad-ba94-252492172b2d-metrics-certs" pod "461caf97-a7d2-47ad-ba94-252492172b2d" (UID: "461caf97-a7d2-47ad-ba94-252492172b2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:06.832389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.832299 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/461caf97-a7d2-47ad-ba94-252492172b2d-metrics-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:06.832389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.832329 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/461caf97-a7d2-47ad-ba94-252492172b2d-openshift-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:06.832389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.832339 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/461caf97-a7d2-47ad-ba94-252492172b2d-maas-api-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:06.832389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.832349 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn8js\" (UniqueName: \"kubernetes.io/projected/461caf97-a7d2-47ad-ba94-252492172b2d-kube-api-access-dn8js\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:06.983294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.983242 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc"] Oct 30 09:29:06.987515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:06.987471 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3de9d-4gvsc"] Oct 30 09:29:07.080728 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:07.080684 5938 generic.go:356] "Generic (PLEG): container finished" podID="461caf97-a7d2-47ad-ba94-252492172b2d" containerID="d6898d8bcfbe8b438baf34477d21ff92217679dc39124982e9c77d0599595370" exitCode=0 Oct 30 09:29:07.080946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:07.080749 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" Oct 30 09:29:07.081067 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:07.081033 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" event={"ID":"461caf97-a7d2-47ad-ba94-252492172b2d","Type":"ContainerDied","Data":"d6898d8bcfbe8b438baf34477d21ff92217679dc39124982e9c77d0599595370"} Oct 30 09:29:07.081193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:07.081101 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l" event={"ID":"461caf97-a7d2-47ad-ba94-252492172b2d","Type":"ContainerDied","Data":"32d15464c1319b14d3c6960f426202bb568246fa4bd9140a8cd6390ec270ee68"} Oct 30 09:29:07.081193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:07.081133 5938 scope.go:122] "RemoveContainer" containerID="d6898d8bcfbe8b438baf34477d21ff92217679dc39124982e9c77d0599595370" Oct 30 09:29:07.101623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:07.101599 5938 scope.go:122] "RemoveContainer" containerID="d6898d8bcfbe8b438baf34477d21ff92217679dc39124982e9c77d0599595370" Oct 30 09:29:07.107669 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:29:07.103165 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6898d8bcfbe8b438baf34477d21ff92217679dc39124982e9c77d0599595370\": container with ID starting with d6898d8bcfbe8b438baf34477d21ff92217679dc39124982e9c77d0599595370 not found: ID does not exist" containerID="d6898d8bcfbe8b438baf34477d21ff92217679dc39124982e9c77d0599595370" Oct 30 09:29:07.107669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:07.103216 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6898d8bcfbe8b438baf34477d21ff92217679dc39124982e9c77d0599595370"} err="failed to get container status \"d6898d8bcfbe8b438baf34477d21ff92217679dc39124982e9c77d0599595370\": rpc error: code = NotFound desc = could not find container \"d6898d8bcfbe8b438baf34477d21ff92217679dc39124982e9c77d0599595370\": container with ID starting with d6898d8bcfbe8b438baf34477d21ff92217679dc39124982e9c77d0599595370 not found: ID does not exist" Oct 30 09:29:07.148178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:07.148137 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l"] Oct 30 09:29:07.152048 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:07.152011 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3de9d-7bc5d5f857-4pn5l"] Oct 30 09:29:07.966447 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:07.966391 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" podUID="b9018a44-392d-46e9-84a3-e62b2da99919" containerName="payload-processing" containerID="cri-o://5527334ae9891b0e10b74e0ee76770968ab407e49bf71fe5c166464a397d4c11" gracePeriod=25 Oct 30 09:29:08.001605 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.001525 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" podUID="93592883-03a8-4dfd-aefb-2ed6ae9ef6cc" containerName="payload-pre-processing" containerID="cri-o://da5067eadb445687a11e61e92b738ef464a643b28ce296333f6935053971d305" gracePeriod=25 Oct 30 09:29:08.089616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.089548 5938 generic.go:356] "Generic (PLEG): container finished" podID="b9018a44-392d-46e9-84a3-e62b2da99919" containerID="5527334ae9891b0e10b74e0ee76770968ab407e49bf71fe5c166464a397d4c11" exitCode=0 Oct 30 09:29:08.089817 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.089651 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" event={"ID":"b9018a44-392d-46e9-84a3-e62b2da99919","Type":"ContainerDied","Data":"5527334ae9891b0e10b74e0ee76770968ab407e49bf71fe5c166464a397d4c11"} Oct 30 09:29:08.261943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.261907 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e571e8-52e5-4916-9c48-e83c8279edac" path="/var/lib/kubelet/pods/29e571e8-52e5-4916-9c48-e83c8279edac/volumes" Oct 30 09:29:08.262599 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.262557 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="461caf97-a7d2-47ad-ba94-252492172b2d" path="/var/lib/kubelet/pods/461caf97-a7d2-47ad-ba94-252492172b2d/volumes" Oct 30 09:29:08.285378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.285350 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" Oct 30 09:29:08.326805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.326775 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" Oct 30 09:29:08.355449 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.355413 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9018a44-392d-46e9-84a3-e62b2da99919-kube-api-access-d2vhh\" (UniqueName: \"kubernetes.io/projected/b9018a44-392d-46e9-84a3-e62b2da99919-kube-api-access-d2vhh\") pod \"b9018a44-392d-46e9-84a3-e62b2da99919\" (UID: \"b9018a44-392d-46e9-84a3-e62b2da99919\") " Oct 30 09:29:08.355710 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.355470 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9018a44-392d-46e9-84a3-e62b2da99919-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9018a44-392d-46e9-84a3-e62b2da99919-plugins-config-volume\") pod \"b9018a44-392d-46e9-84a3-e62b2da99919\" (UID: \"b9018a44-392d-46e9-84a3-e62b2da99919\") " Oct 30 09:29:08.356005 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.355978 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9018a44-392d-46e9-84a3-e62b2da99919-plugins-config-volume" pod "b9018a44-392d-46e9-84a3-e62b2da99919" (UID: "b9018a44-392d-46e9-84a3-e62b2da99919"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:08.358087 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.358055 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9018a44-392d-46e9-84a3-e62b2da99919-kube-api-access-d2vhh" pod "b9018a44-392d-46e9-84a3-e62b2da99919" (UID: "b9018a44-392d-46e9-84a3-e62b2da99919"). InnerVolumeSpecName "kube-api-access-d2vhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:08.457550 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.457512 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93592883-03a8-4dfd-aefb-2ed6ae9ef6cc-kube-api-access-xwnc2\" (UniqueName: \"kubernetes.io/projected/93592883-03a8-4dfd-aefb-2ed6ae9ef6cc-kube-api-access-xwnc2\") pod \"93592883-03a8-4dfd-aefb-2ed6ae9ef6cc\" (UID: \"93592883-03a8-4dfd-aefb-2ed6ae9ef6cc\") " Oct 30 09:29:08.457788 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.457630 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93592883-03a8-4dfd-aefb-2ed6ae9ef6cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93592883-03a8-4dfd-aefb-2ed6ae9ef6cc-plugins-config-volume\") pod \"93592883-03a8-4dfd-aefb-2ed6ae9ef6cc\" (UID: \"93592883-03a8-4dfd-aefb-2ed6ae9ef6cc\") " Oct 30 09:29:08.457950 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.457926 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93592883-03a8-4dfd-aefb-2ed6ae9ef6cc-plugins-config-volume" pod "93592883-03a8-4dfd-aefb-2ed6ae9ef6cc" (UID: "93592883-03a8-4dfd-aefb-2ed6ae9ef6cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:08.458111 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.458094 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9018a44-392d-46e9-84a3-e62b2da99919-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:08.458172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.458116 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93592883-03a8-4dfd-aefb-2ed6ae9ef6cc-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:08.458172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.458130 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2vhh\" (UniqueName: \"kubernetes.io/projected/b9018a44-392d-46e9-84a3-e62b2da99919-kube-api-access-d2vhh\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:08.459894 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.459871 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93592883-03a8-4dfd-aefb-2ed6ae9ef6cc-kube-api-access-xwnc2" pod "93592883-03a8-4dfd-aefb-2ed6ae9ef6cc" (UID: "93592883-03a8-4dfd-aefb-2ed6ae9ef6cc"). InnerVolumeSpecName "kube-api-access-xwnc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:08.559398 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:08.559302 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwnc2\" (UniqueName: \"kubernetes.io/projected/93592883-03a8-4dfd-aefb-2ed6ae9ef6cc-kube-api-access-xwnc2\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:09.100085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:09.100040 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" event={"ID":"b9018a44-392d-46e9-84a3-e62b2da99919","Type":"ContainerDied","Data":"81874bff1a4899000dcd93e91e3e8a8cc1571e4b95a81d1a9dd2bc20c5d51573"} Oct 30 09:29:09.100085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:09.100082 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22" Oct 30 09:29:09.100718 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:09.100162 5938 scope.go:122] "RemoveContainer" containerID="5527334ae9891b0e10b74e0ee76770968ab407e49bf71fe5c166464a397d4c11" Oct 30 09:29:09.101834 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:09.101738 5938 generic.go:356] "Generic (PLEG): container finished" podID="93592883-03a8-4dfd-aefb-2ed6ae9ef6cc" containerID="da5067eadb445687a11e61e92b738ef464a643b28ce296333f6935053971d305" exitCode=0 Oct 30 09:29:09.101834 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:09.101778 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" event={"ID":"93592883-03a8-4dfd-aefb-2ed6ae9ef6cc","Type":"ContainerDied","Data":"da5067eadb445687a11e61e92b738ef464a643b28ce296333f6935053971d305"} Oct 30 09:29:09.101834 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:09.101808 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" Oct 30 09:29:09.102001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:09.101856 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt" event={"ID":"93592883-03a8-4dfd-aefb-2ed6ae9ef6cc","Type":"ContainerDied","Data":"a9a7b2412ee46185a225bda7ec51e101ba584e0286606ca3651e6c4cdf34bd43"} Oct 30 09:29:09.112866 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:09.112846 5938 scope.go:122] "RemoveContainer" containerID="da5067eadb445687a11e61e92b738ef464a643b28ce296333f6935053971d305" Oct 30 09:29:09.123303 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:09.123281 5938 scope.go:122] "RemoveContainer" containerID="da5067eadb445687a11e61e92b738ef464a643b28ce296333f6935053971d305" Oct 30 09:29:09.123721 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:29:09.123697 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5067eadb445687a11e61e92b738ef464a643b28ce296333f6935053971d305\": container with ID starting with da5067eadb445687a11e61e92b738ef464a643b28ce296333f6935053971d305 not found: ID does not exist" containerID="da5067eadb445687a11e61e92b738ef464a643b28ce296333f6935053971d305" Oct 30 09:29:09.123778 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:09.123734 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5067eadb445687a11e61e92b738ef464a643b28ce296333f6935053971d305"} err="failed to get container status \"da5067eadb445687a11e61e92b738ef464a643b28ce296333f6935053971d305\": rpc error: code = NotFound desc = could not find container \"da5067eadb445687a11e61e92b738ef464a643b28ce296333f6935053971d305\": container with ID starting with da5067eadb445687a11e61e92b738ef464a643b28ce296333f6935053971d305 not found: ID does not exist" Oct 30 09:29:09.155679 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:09.155641 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt"] Oct 30 09:29:09.168908 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:09.168863 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cbd96e-57f74fdfd7-sfwvt"] Oct 30 09:29:09.194799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:09.194761 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22"] Oct 30 09:29:09.200806 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:09.200769 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cbd96e-7676b698d-wmn22"] Oct 30 09:29:10.257113 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:10.257078 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93592883-03a8-4dfd-aefb-2ed6ae9ef6cc" path="/var/lib/kubelet/pods/93592883-03a8-4dfd-aefb-2ed6ae9ef6cc/volumes" Oct 30 09:29:10.257530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:10.257503 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9018a44-392d-46e9-84a3-e62b2da99919" path="/var/lib/kubelet/pods/b9018a44-392d-46e9-84a3-e62b2da99919/volumes" Oct 30 09:29:14.113461 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:14.113424 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9"] Oct 30 09:29:14.113946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:14.113735 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" podUID="993fc75c-be0b-49a1-8298-a3dd75e67186" containerName="istio-proxy" containerID="cri-o://afa0ec2ebc8e85c41b953c110dac649af6b24c2c278f46d84a1333f2f40be5a6" gracePeriod=30 Oct 30 09:29:15.837789 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:29:15.837744 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:29:19.396256 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.396174 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:29:19.417471 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.417434 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7"] Oct 30 09:29:19.418041 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.418022 5938 cpu_manager.go:414] "containerMap: removing container" podUID="93592883-03a8-4dfd-aefb-2ed6ae9ef6cc" containerName="payload-pre-processing" Oct 30 09:29:19.418041 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.418041 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93592883-03a8-4dfd-aefb-2ed6ae9ef6cc" containerName="payload-pre-processing" Oct 30 09:29:19.418177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.418057 5938 cpu_manager.go:414] "containerMap: removing container" podUID="993fc75c-be0b-49a1-8298-a3dd75e67186" containerName="istio-proxy" Oct 30 09:29:19.418177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.418062 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993fc75c-be0b-49a1-8298-a3dd75e67186" containerName="istio-proxy" Oct 30 09:29:19.418177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.418099 5938 cpu_manager.go:414] "containerMap: removing container" podUID="461caf97-a7d2-47ad-ba94-252492172b2d" containerName="maas-api" Oct 30 09:29:19.418177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.418104 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461caf97-a7d2-47ad-ba94-252492172b2d" containerName="maas-api" Oct 30 09:29:19.418177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.418117 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b9018a44-392d-46e9-84a3-e62b2da99919" containerName="payload-processing" Oct 30 09:29:19.418177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.418127 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9018a44-392d-46e9-84a3-e62b2da99919" containerName="payload-processing" Oct 30 09:29:19.418177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.418145 5938 cpu_manager.go:414] "containerMap: removing container" podUID="29e571e8-52e5-4916-9c48-e83c8279edac" containerName="revoke-keys" Oct 30 09:29:19.418177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.418151 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e571e8-52e5-4916-9c48-e83c8279edac" containerName="revoke-keys" Oct 30 09:29:19.418420 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.418236 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="461caf97-a7d2-47ad-ba94-252492172b2d" containerName="maas-api" Oct 30 09:29:19.418420 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.418246 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9018a44-392d-46e9-84a3-e62b2da99919" containerName="payload-processing" Oct 30 09:29:19.418420 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.418255 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e571e8-52e5-4916-9c48-e83c8279edac" containerName="revoke-keys" Oct 30 09:29:19.418420 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.418262 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="93592883-03a8-4dfd-aefb-2ed6ae9ef6cc" containerName="payload-pre-processing" Oct 30 09:29:19.418420 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.418268 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="993fc75c-be0b-49a1-8298-a3dd75e67186" containerName="istio-proxy" Oct 30 09:29:19.421697 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.421678 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7" Oct 30 09:29:19.438022 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.437990 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:29:19.438022 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.438007 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:29:19.440490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.439940 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7"] Oct 30 09:29:19.478165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.478121 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993fc75c-be0b-49a1-8298-a3dd75e67186-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993fc75c-be0b-49a1-8298-a3dd75e67186-istiod-ca-cert\") pod \"993fc75c-be0b-49a1-8298-a3dd75e67186\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " Oct 30 09:29:19.478337 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.478204 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-workload-socket\") pod \"993fc75c-be0b-49a1-8298-a3dd75e67186\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " Oct 30 09:29:19.478337 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.478329 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-podinfo\") pod \"993fc75c-be0b-49a1-8298-a3dd75e67186\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " Oct 30 09:29:19.478437 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.478377 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-credential-socket\") pod \"993fc75c-be0b-49a1-8298-a3dd75e67186\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " Oct 30 09:29:19.478486 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.478471 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993fc75c-be0b-49a1-8298-a3dd75e67186-kube-api-access-2pbhn\" (UniqueName: \"kubernetes.io/projected/993fc75c-be0b-49a1-8298-a3dd75e67186-kube-api-access-2pbhn\") pod \"993fc75c-be0b-49a1-8298-a3dd75e67186\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " Oct 30 09:29:19.478528 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.478508 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-data\") pod \"993fc75c-be0b-49a1-8298-a3dd75e67186\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " Oct 30 09:29:19.478564 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.478553 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-workload-certs\") pod \"993fc75c-be0b-49a1-8298-a3dd75e67186\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " Oct 30 09:29:19.478654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.478640 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-token\" (UniqueName: \"kubernetes.io/projected/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-token\") pod \"993fc75c-be0b-49a1-8298-a3dd75e67186\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " Oct 30 09:29:19.478728 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.478714 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-envoy\") pod \"993fc75c-be0b-49a1-8298-a3dd75e67186\" (UID: \"993fc75c-be0b-49a1-8298-a3dd75e67186\") " Oct 30 09:29:19.480494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.480461 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc"] Oct 30 09:29:19.480793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.480759 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" podUID="00c9d38a-98f0-4598-8650-d49c9f16be07" containerName="istio-proxy" containerID="cri-o://06753652c39f2159a69e910c5dd6fdf02c976f54b7f8325680e2cb8c129db0a8" gracePeriod=30 Oct 30 09:29:19.481056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.481030 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993fc75c-be0b-49a1-8298-a3dd75e67186-istiod-ca-cert" pod "993fc75c-be0b-49a1-8298-a3dd75e67186" (UID: "993fc75c-be0b-49a1-8298-a3dd75e67186"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:19.481763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.481308 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-workload-socket" pod "993fc75c-be0b-49a1-8298-a3dd75e67186" (UID: "993fc75c-be0b-49a1-8298-a3dd75e67186"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:19.481763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.481489 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-data" pod "993fc75c-be0b-49a1-8298-a3dd75e67186" (UID: "993fc75c-be0b-49a1-8298-a3dd75e67186"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:19.481763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.481670 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-workload-certs" pod "993fc75c-be0b-49a1-8298-a3dd75e67186" (UID: "993fc75c-be0b-49a1-8298-a3dd75e67186"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:19.482197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.482171 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-credential-socket" pod "993fc75c-be0b-49a1-8298-a3dd75e67186" (UID: "993fc75c-be0b-49a1-8298-a3dd75e67186"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:19.482657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.482581 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-envoy" pod "993fc75c-be0b-49a1-8298-a3dd75e67186" (UID: "993fc75c-be0b-49a1-8298-a3dd75e67186"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:19.485018 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.484986 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-token" pod "993fc75c-be0b-49a1-8298-a3dd75e67186" (UID: "993fc75c-be0b-49a1-8298-a3dd75e67186"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:19.485166 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.485136 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993fc75c-be0b-49a1-8298-a3dd75e67186-kube-api-access-2pbhn" pod "993fc75c-be0b-49a1-8298-a3dd75e67186" (UID: "993fc75c-be0b-49a1-8298-a3dd75e67186"). InnerVolumeSpecName "kube-api-access-2pbhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:19.486509 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.486485 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-podinfo" pod "993fc75c-be0b-49a1-8298-a3dd75e67186" (UID: "993fc75c-be0b-49a1-8298-a3dd75e67186"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:29:19.580255 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.580216 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7631eda-702a-4552-9fa4-cae2275317d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d639a65f-f6bx7\" (UID: \"e7631eda-702a-4552-9fa4-cae2275317d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7" Oct 30 09:29:19.580454 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.580422 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps885\" (UniqueName: \"kubernetes.io/projected/e7631eda-702a-4552-9fa4-cae2275317d3-kube-api-access-ps885\") pod \"maas-api-revoke-sub-d639a65f-f6bx7\" (UID: \"e7631eda-702a-4552-9fa4-cae2275317d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7" Oct 30 09:29:19.580502 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.580483 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7631eda-702a-4552-9fa4-cae2275317d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d639a65f-f6bx7\" (UID: \"e7631eda-702a-4552-9fa4-cae2275317d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7" Oct 30 09:29:19.580659 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.580643 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.580700 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.580666 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993fc75c-be0b-49a1-8298-a3dd75e67186-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.580700 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.580678 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.580700 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.580687 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.580700 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.580695 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.580824 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.580705 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pbhn\" (UniqueName: \"kubernetes.io/projected/993fc75c-be0b-49a1-8298-a3dd75e67186-kube-api-access-2pbhn\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.580824 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.580715 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.580824 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.580725 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993fc75c-be0b-49a1-8298-a3dd75e67186-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.580824 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.580739 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/993fc75c-be0b-49a1-8298-a3dd75e67186-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.682301 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.682202 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7631eda-702a-4552-9fa4-cae2275317d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d639a65f-f6bx7\" (UID: \"e7631eda-702a-4552-9fa4-cae2275317d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7" Oct 30 09:29:19.682467 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.682318 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps885\" (UniqueName: \"kubernetes.io/projected/e7631eda-702a-4552-9fa4-cae2275317d3-kube-api-access-ps885\") pod \"maas-api-revoke-sub-d639a65f-f6bx7\" (UID: \"e7631eda-702a-4552-9fa4-cae2275317d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7" Oct 30 09:29:19.682467 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.682360 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7631eda-702a-4552-9fa4-cae2275317d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d639a65f-f6bx7\" (UID: \"e7631eda-702a-4552-9fa4-cae2275317d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7" Oct 30 09:29:19.682962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.682938 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7631eda-702a-4552-9fa4-cae2275317d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d639a65f-f6bx7\" (UID: \"e7631eda-702a-4552-9fa4-cae2275317d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7" Oct 30 09:29:19.685096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.685069 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7631eda-702a-4552-9fa4-cae2275317d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d639a65f-f6bx7\" (UID: \"e7631eda-702a-4552-9fa4-cae2275317d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7" Oct 30 09:29:19.704297 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.704262 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps885\" (UniqueName: \"kubernetes.io/projected/e7631eda-702a-4552-9fa4-cae2275317d3-kube-api-access-ps885\") pod \"maas-api-revoke-sub-d639a65f-f6bx7\" (UID: \"e7631eda-702a-4552-9fa4-cae2275317d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7" Oct 30 09:29:19.735869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:19.735828 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7" Oct 30 09:29:20.040950 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:20.040922 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7"] Oct 30 09:29:20.042966 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:29:20.042938 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7631eda_702a_4552_9fa4_cae2275317d3.slice/crio-b7ac410458d004b28a093bf3cae6a5228bc5ca646b874ea499b55ac66f90b1fc WatchSource:0}: Error finding container b7ac410458d004b28a093bf3cae6a5228bc5ca646b874ea499b55ac66f90b1fc: Status 404 returned error can't find the container with id b7ac410458d004b28a093bf3cae6a5228bc5ca646b874ea499b55ac66f90b1fc Oct 30 09:29:20.185809 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:20.185770 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7" event={"ID":"e7631eda-702a-4552-9fa4-cae2275317d3","Type":"ContainerStarted","Data":"b7ac410458d004b28a093bf3cae6a5228bc5ca646b874ea499b55ac66f90b1fc"} Oct 30 09:29:20.187235 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:20.187198 5938 generic.go:356] "Generic (PLEG): container finished" podID="993fc75c-be0b-49a1-8298-a3dd75e67186" containerID="afa0ec2ebc8e85c41b953c110dac649af6b24c2c278f46d84a1333f2f40be5a6" exitCode=0 Oct 30 09:29:20.187411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:20.187234 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" event={"ID":"993fc75c-be0b-49a1-8298-a3dd75e67186","Type":"ContainerDied","Data":"afa0ec2ebc8e85c41b953c110dac649af6b24c2c278f46d84a1333f2f40be5a6"} Oct 30 09:29:20.187411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:20.187266 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" Oct 30 09:29:20.187411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:20.187290 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9" event={"ID":"993fc75c-be0b-49a1-8298-a3dd75e67186","Type":"ContainerDied","Data":"eaf7ea0fc7a8e0cb4d30cfbbc61e2aa9d11c8c3b90ae2f9a51cc658688b3dd52"} Oct 30 09:29:20.187411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:20.187317 5938 scope.go:122] "RemoveContainer" containerID="afa0ec2ebc8e85c41b953c110dac649af6b24c2c278f46d84a1333f2f40be5a6" Oct 30 09:29:20.199051 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:20.199026 5938 scope.go:122] "RemoveContainer" containerID="afa0ec2ebc8e85c41b953c110dac649af6b24c2c278f46d84a1333f2f40be5a6" Oct 30 09:29:20.199432 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:29:20.199400 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa0ec2ebc8e85c41b953c110dac649af6b24c2c278f46d84a1333f2f40be5a6\": container with ID starting with afa0ec2ebc8e85c41b953c110dac649af6b24c2c278f46d84a1333f2f40be5a6 not found: ID does not exist" containerID="afa0ec2ebc8e85c41b953c110dac649af6b24c2c278f46d84a1333f2f40be5a6" Oct 30 09:29:20.199524 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:20.199444 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa0ec2ebc8e85c41b953c110dac649af6b24c2c278f46d84a1333f2f40be5a6"} err="failed to get container status \"afa0ec2ebc8e85c41b953c110dac649af6b24c2c278f46d84a1333f2f40be5a6\": rpc error: code = NotFound desc = could not find container \"afa0ec2ebc8e85c41b953c110dac649af6b24c2c278f46d84a1333f2f40be5a6\": container with ID starting with afa0ec2ebc8e85c41b953c110dac649af6b24c2c278f46d84a1333f2f40be5a6 not found: ID does not exist" Oct 30 09:29:20.244750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:20.244677 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9"] Oct 30 09:29:20.259180 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:20.259140 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-rwqz9"] Oct 30 09:29:21.196459 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:21.196361 5938 generic.go:356] "Generic (PLEG): container finished" podID="e7631eda-702a-4552-9fa4-cae2275317d3" containerID="d510acf079f385b28bff3b3c93031c9688f363a80a5d4264fe4ee31f7d390b63" exitCode=0 Oct 30 09:29:21.196459 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:21.196421 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7" event={"ID":"e7631eda-702a-4552-9fa4-cae2275317d3","Type":"ContainerDied","Data":"d510acf079f385b28bff3b3c93031c9688f363a80a5d4264fe4ee31f7d390b63"} Oct 30 09:29:22.259016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:22.258982 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993fc75c-be0b-49a1-8298-a3dd75e67186" path="/var/lib/kubelet/pods/993fc75c-be0b-49a1-8298-a3dd75e67186/volumes" Oct 30 09:29:22.366903 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:22.366875 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7" Oct 30 09:29:22.531167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:22.531056 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7631eda-702a-4552-9fa4-cae2275317d3-kube-api-access-ps885\" (UniqueName: \"kubernetes.io/projected/e7631eda-702a-4552-9fa4-cae2275317d3-kube-api-access-ps885\") pod \"e7631eda-702a-4552-9fa4-cae2275317d3\" (UID: \"e7631eda-702a-4552-9fa4-cae2275317d3\") " Oct 30 09:29:22.531167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:22.531163 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7631eda-702a-4552-9fa4-cae2275317d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7631eda-702a-4552-9fa4-cae2275317d3-maas-api-openshift-ca\") pod \"e7631eda-702a-4552-9fa4-cae2275317d3\" (UID: \"e7631eda-702a-4552-9fa4-cae2275317d3\") " Oct 30 09:29:22.531420 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:22.531400 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7631eda-702a-4552-9fa4-cae2275317d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7631eda-702a-4552-9fa4-cae2275317d3-maas-api-xks-ca\") pod \"e7631eda-702a-4552-9fa4-cae2275317d3\" (UID: \"e7631eda-702a-4552-9fa4-cae2275317d3\") " Oct 30 09:29:22.531535 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:22.531497 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7631eda-702a-4552-9fa4-cae2275317d3-maas-api-openshift-ca" pod "e7631eda-702a-4552-9fa4-cae2275317d3" (UID: "e7631eda-702a-4552-9fa4-cae2275317d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:22.531886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:22.531863 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7631eda-702a-4552-9fa4-cae2275317d3-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:22.533835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:22.533805 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7631eda-702a-4552-9fa4-cae2275317d3-maas-api-xks-ca" pod "e7631eda-702a-4552-9fa4-cae2275317d3" (UID: "e7631eda-702a-4552-9fa4-cae2275317d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:22.533957 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:22.533923 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7631eda-702a-4552-9fa4-cae2275317d3-kube-api-access-ps885" pod "e7631eda-702a-4552-9fa4-cae2275317d3" (UID: "e7631eda-702a-4552-9fa4-cae2275317d3"). InnerVolumeSpecName "kube-api-access-ps885". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:22.633332 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:22.633291 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps885\" (UniqueName: \"kubernetes.io/projected/e7631eda-702a-4552-9fa4-cae2275317d3-kube-api-access-ps885\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:22.633332 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:22.633332 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7631eda-702a-4552-9fa4-cae2275317d3-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:23.210762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:23.210723 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7" event={"ID":"e7631eda-702a-4552-9fa4-cae2275317d3","Type":"ContainerDied","Data":"b7ac410458d004b28a093bf3cae6a5228bc5ca646b874ea499b55ac66f90b1fc"} Oct 30 09:29:23.210762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:23.210752 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7" Oct 30 09:29:23.210987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:23.210788 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7ac410458d004b28a093bf3cae6a5228bc5ca646b874ea499b55ac66f90b1fc" Oct 30 09:29:24.840433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.840393 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:29:24.870604 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.870559 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-workload-socket\") pod \"00c9d38a-98f0-4598-8650-d49c9f16be07\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " Oct 30 09:29:24.870787 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.870666 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-data\") pod \"00c9d38a-98f0-4598-8650-d49c9f16be07\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " Oct 30 09:29:24.870787 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.870721 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c9d38a-98f0-4598-8650-d49c9f16be07-kube-api-access-cfzmp\" (UniqueName: \"kubernetes.io/projected/00c9d38a-98f0-4598-8650-d49c9f16be07-kube-api-access-cfzmp\") pod \"00c9d38a-98f0-4598-8650-d49c9f16be07\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " Oct 30 09:29:24.870787 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.870772 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-token\" (UniqueName: \"kubernetes.io/projected/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-token\") pod \"00c9d38a-98f0-4598-8650-d49c9f16be07\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " Oct 30 09:29:24.870906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.870827 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-workload-certs\") pod \"00c9d38a-98f0-4598-8650-d49c9f16be07\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " Oct 30 09:29:24.870960 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.870903 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-workload-socket" pod "00c9d38a-98f0-4598-8650-d49c9f16be07" (UID: "00c9d38a-98f0-4598-8650-d49c9f16be07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:24.870960 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.870944 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-envoy\") pod \"00c9d38a-98f0-4598-8650-d49c9f16be07\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " Oct 30 09:29:24.871069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.870982 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-data" pod "00c9d38a-98f0-4598-8650-d49c9f16be07" (UID: "00c9d38a-98f0-4598-8650-d49c9f16be07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:24.871069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.871044 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c9d38a-98f0-4598-8650-d49c9f16be07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c9d38a-98f0-4598-8650-d49c9f16be07-istiod-ca-cert\") pod \"00c9d38a-98f0-4598-8650-d49c9f16be07\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " Oct 30 09:29:24.871179 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.871083 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-podinfo\") pod \"00c9d38a-98f0-4598-8650-d49c9f16be07\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " Oct 30 09:29:24.871235 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.871190 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-credential-socket\") pod \"00c9d38a-98f0-4598-8650-d49c9f16be07\" (UID: \"00c9d38a-98f0-4598-8650-d49c9f16be07\") " Oct 30 09:29:24.871235 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.871186 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-workload-certs" pod "00c9d38a-98f0-4598-8650-d49c9f16be07" (UID: "00c9d38a-98f0-4598-8650-d49c9f16be07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:24.871481 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.871451 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c9d38a-98f0-4598-8650-d49c9f16be07-istiod-ca-cert" pod "00c9d38a-98f0-4598-8650-d49c9f16be07" (UID: "00c9d38a-98f0-4598-8650-d49c9f16be07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:24.871566 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.871462 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-credential-socket" pod "00c9d38a-98f0-4598-8650-d49c9f16be07" (UID: "00c9d38a-98f0-4598-8650-d49c9f16be07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:24.872013 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.871987 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:24.872139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.872045 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c9d38a-98f0-4598-8650-d49c9f16be07-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:24.872139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.872061 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:24.872139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.872075 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:24.872139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.872094 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:24.873376 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.873339 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-token" pod "00c9d38a-98f0-4598-8650-d49c9f16be07" (UID: "00c9d38a-98f0-4598-8650-d49c9f16be07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:24.873474 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.873387 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-envoy" pod "00c9d38a-98f0-4598-8650-d49c9f16be07" (UID: "00c9d38a-98f0-4598-8650-d49c9f16be07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:24.873474 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.873390 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c9d38a-98f0-4598-8650-d49c9f16be07-kube-api-access-cfzmp" pod "00c9d38a-98f0-4598-8650-d49c9f16be07" (UID: "00c9d38a-98f0-4598-8650-d49c9f16be07"). InnerVolumeSpecName "kube-api-access-cfzmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:24.873615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.873539 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-podinfo" pod "00c9d38a-98f0-4598-8650-d49c9f16be07" (UID: "00c9d38a-98f0-4598-8650-d49c9f16be07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:29:24.973520 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.973475 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfzmp\" (UniqueName: \"kubernetes.io/projected/00c9d38a-98f0-4598-8650-d49c9f16be07-kube-api-access-cfzmp\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:24.973520 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.973520 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:24.973520 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.973535 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:24.973828 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:24.973544 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c9d38a-98f0-4598-8650-d49c9f16be07-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:25.106972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.106876 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8"] Oct 30 09:29:25.107447 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.107433 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e7631eda-702a-4552-9fa4-cae2275317d3" containerName="revoke-keys" Oct 30 09:29:25.107498 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.107449 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7631eda-702a-4552-9fa4-cae2275317d3" containerName="revoke-keys" Oct 30 09:29:25.107498 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.107489 5938 cpu_manager.go:414] "containerMap: removing container" podUID="00c9d38a-98f0-4598-8650-d49c9f16be07" containerName="istio-proxy" Oct 30 09:29:25.107498 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.107494 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c9d38a-98f0-4598-8650-d49c9f16be07" containerName="istio-proxy" Oct 30 09:29:25.107662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.107651 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7631eda-702a-4552-9fa4-cae2275317d3" containerName="revoke-keys" Oct 30 09:29:25.107707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.107676 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c9d38a-98f0-4598-8650-d49c9f16be07" containerName="istio-proxy" Oct 30 09:29:25.111305 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.111285 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" Oct 30 09:29:25.127797 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.127750 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:29:25.127997 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.127856 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:29:25.129874 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.129844 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8"] Oct 30 09:29:25.176218 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.176184 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f0797f-5874-4771-9b53-173dcf49b89f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8\" (UID: \"b3f0797f-5874-4771-9b53-173dcf49b89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" Oct 30 09:29:25.176380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.176350 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f0797f-5874-4771-9b53-173dcf49b89f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8\" (UID: \"b3f0797f-5874-4771-9b53-173dcf49b89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" Oct 30 09:29:25.176493 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.176480 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lndg\" (UniqueName: \"kubernetes.io/projected/b3f0797f-5874-4771-9b53-173dcf49b89f-kube-api-access-5lndg\") pod \"maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8\" (UID: \"b3f0797f-5874-4771-9b53-173dcf49b89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" Oct 30 09:29:25.225118 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.225086 5938 generic.go:356] "Generic (PLEG): container finished" podID="00c9d38a-98f0-4598-8650-d49c9f16be07" containerID="06753652c39f2159a69e910c5dd6fdf02c976f54b7f8325680e2cb8c129db0a8" exitCode=0 Oct 30 09:29:25.225311 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.225140 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" Oct 30 09:29:25.225311 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.225142 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" event={"ID":"00c9d38a-98f0-4598-8650-d49c9f16be07","Type":"ContainerDied","Data":"06753652c39f2159a69e910c5dd6fdf02c976f54b7f8325680e2cb8c129db0a8"} Oct 30 09:29:25.225311 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.225269 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc" event={"ID":"00c9d38a-98f0-4598-8650-d49c9f16be07","Type":"ContainerDied","Data":"36d2d5dc86183a921f6d917f44cd71bd95bf31a42941b315c6d5d3265b8887fa"} Oct 30 09:29:25.225311 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.225304 5938 scope.go:122] "RemoveContainer" containerID="06753652c39f2159a69e910c5dd6fdf02c976f54b7f8325680e2cb8c129db0a8" Oct 30 09:29:25.236994 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.236973 5938 scope.go:122] "RemoveContainer" containerID="06753652c39f2159a69e910c5dd6fdf02c976f54b7f8325680e2cb8c129db0a8" Oct 30 09:29:25.237306 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:29:25.237288 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06753652c39f2159a69e910c5dd6fdf02c976f54b7f8325680e2cb8c129db0a8\": container with ID starting with 06753652c39f2159a69e910c5dd6fdf02c976f54b7f8325680e2cb8c129db0a8 not found: ID does not exist" containerID="06753652c39f2159a69e910c5dd6fdf02c976f54b7f8325680e2cb8c129db0a8" Oct 30 09:29:25.237370 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.237317 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06753652c39f2159a69e910c5dd6fdf02c976f54b7f8325680e2cb8c129db0a8"} err="failed to get container status \"06753652c39f2159a69e910c5dd6fdf02c976f54b7f8325680e2cb8c129db0a8\": rpc error: code = NotFound desc = could not find container \"06753652c39f2159a69e910c5dd6fdf02c976f54b7f8325680e2cb8c129db0a8\": container with ID starting with 06753652c39f2159a69e910c5dd6fdf02c976f54b7f8325680e2cb8c129db0a8 not found: ID does not exist" Oct 30 09:29:25.283752 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.283705 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f0797f-5874-4771-9b53-173dcf49b89f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8\" (UID: \"b3f0797f-5874-4771-9b53-173dcf49b89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" Oct 30 09:29:25.283976 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.283881 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lndg\" (UniqueName: \"kubernetes.io/projected/b3f0797f-5874-4771-9b53-173dcf49b89f-kube-api-access-5lndg\") pod \"maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8\" (UID: \"b3f0797f-5874-4771-9b53-173dcf49b89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" Oct 30 09:29:25.283976 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.283970 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f0797f-5874-4771-9b53-173dcf49b89f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8\" (UID: \"b3f0797f-5874-4771-9b53-173dcf49b89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" Oct 30 09:29:25.285133 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.285085 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f0797f-5874-4771-9b53-173dcf49b89f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8\" (UID: \"b3f0797f-5874-4771-9b53-173dcf49b89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" Oct 30 09:29:25.287657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.287629 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f0797f-5874-4771-9b53-173dcf49b89f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8\" (UID: \"b3f0797f-5874-4771-9b53-173dcf49b89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" Oct 30 09:29:25.287966 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.287937 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc"] Oct 30 09:29:25.291685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.291655 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-4jmrc"] Oct 30 09:29:25.306692 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.306656 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lndg\" (UniqueName: \"kubernetes.io/projected/b3f0797f-5874-4771-9b53-173dcf49b89f-kube-api-access-5lndg\") pod \"maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8\" (UID: \"b3f0797f-5874-4771-9b53-173dcf49b89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" Oct 30 09:29:25.425839 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.425797 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" Oct 30 09:29:25.733924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:25.733875 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8"] Oct 30 09:29:25.735513 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:29:25.735476 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f0797f_5874_4771_9b53_173dcf49b89f.slice/crio-931cdb5377eab7adcac3a4c5c02478c2180c2ff40ad5552c6cce6edb8f35e895 WatchSource:0}: Error finding container 931cdb5377eab7adcac3a4c5c02478c2180c2ff40ad5552c6cce6edb8f35e895: Status 404 returned error can't find the container with id 931cdb5377eab7adcac3a4c5c02478c2180c2ff40ad5552c6cce6edb8f35e895 Oct 30 09:29:26.235390 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:26.235346 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" event={"ID":"b3f0797f-5874-4771-9b53-173dcf49b89f","Type":"ContainerStarted","Data":"89289d2ec29d267cac8f4b3bebbf52ef977b8010d36da7f0b28aa2007dff86da"} Oct 30 09:29:26.235929 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:26.235417 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" event={"ID":"b3f0797f-5874-4771-9b53-173dcf49b89f","Type":"ContainerStarted","Data":"931cdb5377eab7adcac3a4c5c02478c2180c2ff40ad5552c6cce6edb8f35e895"} Oct 30 09:29:26.258399 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:26.258364 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c9d38a-98f0-4598-8650-d49c9f16be07" path="/var/lib/kubelet/pods/00c9d38a-98f0-4598-8650-d49c9f16be07/volumes" Oct 30 09:29:26.281154 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:26.281028 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" podStartSLOduration=1.2809946220000001 podStartE2EDuration="1.280994622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:26.279766066 +0000 UTC m=+2084.142471887" watchObservedRunningTime="2026-10-30 09:29:26.280994622 +0000 UTC m=+2084.143700443" Oct 30 09:29:26.681515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:26.681473 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr"] Oct 30 09:29:26.681797 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:26.681771 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" podUID="13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" containerName="istio-proxy" containerID="cri-o://8e40ec740a8035e020b1fd43aca281322226955dcbb2cf9e3bf125d27c7485e1" gracePeriod=30 Oct 30 09:29:27.243927 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:27.243888 5938 generic.go:356] "Generic (PLEG): container finished" podID="b3f0797f-5874-4771-9b53-173dcf49b89f" containerID="89289d2ec29d267cac8f4b3bebbf52ef977b8010d36da7f0b28aa2007dff86da" exitCode=0 Oct 30 09:29:27.244353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:27.243981 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" event={"ID":"b3f0797f-5874-4771-9b53-173dcf49b89f","Type":"ContainerDied","Data":"89289d2ec29d267cac8f4b3bebbf52ef977b8010d36da7f0b28aa2007dff86da"} Oct 30 09:29:28.413237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.413208 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" Oct 30 09:29:28.528354 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.528259 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3f0797f-5874-4771-9b53-173dcf49b89f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f0797f-5874-4771-9b53-173dcf49b89f-maas-api-xks-ca\") pod \"b3f0797f-5874-4771-9b53-173dcf49b89f\" (UID: \"b3f0797f-5874-4771-9b53-173dcf49b89f\") " Oct 30 09:29:28.528518 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.528421 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3f0797f-5874-4771-9b53-173dcf49b89f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f0797f-5874-4771-9b53-173dcf49b89f-maas-api-openshift-ca\") pod \"b3f0797f-5874-4771-9b53-173dcf49b89f\" (UID: \"b3f0797f-5874-4771-9b53-173dcf49b89f\") " Oct 30 09:29:28.528642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.528574 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f0797f-5874-4771-9b53-173dcf49b89f-kube-api-access-5lndg\" (UniqueName: \"kubernetes.io/projected/b3f0797f-5874-4771-9b53-173dcf49b89f-kube-api-access-5lndg\") pod \"b3f0797f-5874-4771-9b53-173dcf49b89f\" (UID: \"b3f0797f-5874-4771-9b53-173dcf49b89f\") " Oct 30 09:29:28.528872 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.528844 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f0797f-5874-4771-9b53-173dcf49b89f-maas-api-openshift-ca" pod "b3f0797f-5874-4771-9b53-173dcf49b89f" (UID: "b3f0797f-5874-4771-9b53-173dcf49b89f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:28.529110 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.529093 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f0797f-5874-4771-9b53-173dcf49b89f-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:28.530500 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.530467 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3f0797f-5874-4771-9b53-173dcf49b89f-maas-api-xks-ca" pod "b3f0797f-5874-4771-9b53-173dcf49b89f" (UID: "b3f0797f-5874-4771-9b53-173dcf49b89f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:28.530801 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.530785 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f0797f-5874-4771-9b53-173dcf49b89f-kube-api-access-5lndg" pod "b3f0797f-5874-4771-9b53-173dcf49b89f" (UID: "b3f0797f-5874-4771-9b53-173dcf49b89f"). InnerVolumeSpecName "kube-api-access-5lndg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:28.582169 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.582113 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj"] Oct 30 09:29:28.583661 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.583629 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b3f0797f-5874-4771-9b53-173dcf49b89f" containerName="revoke-keys" Oct 30 09:29:28.583661 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.583659 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f0797f-5874-4771-9b53-173dcf49b89f" containerName="revoke-keys" Oct 30 09:29:28.583966 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.583950 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f0797f-5874-4771-9b53-173dcf49b89f" containerName="revoke-keys" Oct 30 09:29:28.589440 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.589410 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj" Oct 30 09:29:28.606100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.603470 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj"] Oct 30 09:29:28.630934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.630886 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba44b3e-lbxrj\" (UID: \"ddd86d9a-04a2-40f3-b74c-2b92781a0b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj" Oct 30 09:29:28.631148 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.631013 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25dh6\" (UniqueName: \"kubernetes.io/projected/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-kube-api-access-25dh6\") pod \"maas-api-revoke-sub-fba44b3e-lbxrj\" (UID: \"ddd86d9a-04a2-40f3-b74c-2b92781a0b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj" Oct 30 09:29:28.631227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.631213 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba44b3e-lbxrj\" (UID: \"ddd86d9a-04a2-40f3-b74c-2b92781a0b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj" Oct 30 09:29:28.631436 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.631413 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lndg\" (UniqueName: \"kubernetes.io/projected/b3f0797f-5874-4771-9b53-173dcf49b89f-kube-api-access-5lndg\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:28.631494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.631443 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f0797f-5874-4771-9b53-173dcf49b89f-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:28.733090 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.733044 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba44b3e-lbxrj\" (UID: \"ddd86d9a-04a2-40f3-b74c-2b92781a0b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj" Oct 30 09:29:28.733312 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.733128 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25dh6\" (UniqueName: \"kubernetes.io/projected/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-kube-api-access-25dh6\") pod \"maas-api-revoke-sub-fba44b3e-lbxrj\" (UID: \"ddd86d9a-04a2-40f3-b74c-2b92781a0b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj" Oct 30 09:29:28.733312 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.733227 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba44b3e-lbxrj\" (UID: \"ddd86d9a-04a2-40f3-b74c-2b92781a0b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj" Oct 30 09:29:28.733883 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.733855 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba44b3e-lbxrj\" (UID: \"ddd86d9a-04a2-40f3-b74c-2b92781a0b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj" Oct 30 09:29:28.735835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.735817 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba44b3e-lbxrj\" (UID: \"ddd86d9a-04a2-40f3-b74c-2b92781a0b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj" Oct 30 09:29:28.754148 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.754115 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25dh6\" (UniqueName: \"kubernetes.io/projected/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-kube-api-access-25dh6\") pod \"maas-api-revoke-sub-fba44b3e-lbxrj\" (UID: \"ddd86d9a-04a2-40f3-b74c-2b92781a0b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj" Oct 30 09:29:28.919613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:28.918132 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj" Oct 30 09:29:29.236176 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:29.236146 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj"] Oct 30 09:29:29.258553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:29.258519 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj" event={"ID":"ddd86d9a-04a2-40f3-b74c-2b92781a0b59","Type":"ContainerStarted","Data":"e2005708f6fa338be39829da4876f7a21c7bb63dfbc5d64ea888e5952b2e2e9e"} Oct 30 09:29:29.260240 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:29.260213 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" Oct 30 09:29:29.260385 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:29.260229 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8" event={"ID":"b3f0797f-5874-4771-9b53-173dcf49b89f","Type":"ContainerDied","Data":"931cdb5377eab7adcac3a4c5c02478c2180c2ff40ad5552c6cce6edb8f35e895"} Oct 30 09:29:29.260385 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:29.260292 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="931cdb5377eab7adcac3a4c5c02478c2180c2ff40ad5552c6cce6edb8f35e895" Oct 30 09:29:30.268907 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:30.268872 5938 generic.go:356] "Generic (PLEG): container finished" podID="ddd86d9a-04a2-40f3-b74c-2b92781a0b59" containerID="40eb453bfda17f20725b8f43cdb8e96b6e2dcee9bdb8a2054d70d8218769e9c2" exitCode=0 Oct 30 09:29:30.269320 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:30.268934 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj" event={"ID":"ddd86d9a-04a2-40f3-b74c-2b92781a0b59","Type":"ContainerDied","Data":"40eb453bfda17f20725b8f43cdb8e96b6e2dcee9bdb8a2054d70d8218769e9c2"} Oct 30 09:29:31.536669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:31.531392 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj" Oct 30 09:29:31.568610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:31.568472 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-kube-api-access-25dh6\" (UniqueName: \"kubernetes.io/projected/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-kube-api-access-25dh6\") pod \"ddd86d9a-04a2-40f3-b74c-2b92781a0b59\" (UID: \"ddd86d9a-04a2-40f3-b74c-2b92781a0b59\") " Oct 30 09:29:31.568814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:31.568691 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-maas-api-xks-ca\") pod \"ddd86d9a-04a2-40f3-b74c-2b92781a0b59\" (UID: \"ddd86d9a-04a2-40f3-b74c-2b92781a0b59\") " Oct 30 09:29:31.571657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:31.569110 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-maas-api-openshift-ca\") pod \"ddd86d9a-04a2-40f3-b74c-2b92781a0b59\" (UID: \"ddd86d9a-04a2-40f3-b74c-2b92781a0b59\") " Oct 30 09:29:31.571657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:31.570767 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-maas-api-openshift-ca" pod "ddd86d9a-04a2-40f3-b74c-2b92781a0b59" (UID: "ddd86d9a-04a2-40f3-b74c-2b92781a0b59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:31.584864 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:31.584052 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-maas-api-xks-ca" pod "ddd86d9a-04a2-40f3-b74c-2b92781a0b59" (UID: "ddd86d9a-04a2-40f3-b74c-2b92781a0b59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:31.591524 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:31.591284 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-kube-api-access-25dh6" pod "ddd86d9a-04a2-40f3-b74c-2b92781a0b59" (UID: "ddd86d9a-04a2-40f3-b74c-2b92781a0b59"). InnerVolumeSpecName "kube-api-access-25dh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:31.677614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:31.677239 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25dh6\" (UniqueName: \"kubernetes.io/projected/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-kube-api-access-25dh6\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:31.677614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:31.677273 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:31.677614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:31.677288 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd86d9a-04a2-40f3-b74c-2b92781a0b59-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.001338 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.001308 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:29:32.089387 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.089280 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-workload-certs\") pod \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " Oct 30 09:29:32.089557 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.089391 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-kube-api-access-s576l\" (UniqueName: \"kubernetes.io/projected/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-kube-api-access-s576l\") pod \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " Oct 30 09:29:32.089557 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.089543 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-podinfo\") pod \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " Oct 30 09:29:32.089737 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.089637 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istiod-ca-cert\") pod \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " Oct 30 09:29:32.089737 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.089646 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-workload-certs" pod "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" (UID: "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:32.089737 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.089732 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-envoy\") pod \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " Oct 30 09:29:32.089902 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.089773 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-credential-socket\") pod \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " Oct 30 09:29:32.089902 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.089807 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-workload-socket\") pod \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " Oct 30 09:29:32.090009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.089862 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-data\") pod \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " Oct 30 09:29:32.090009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.089956 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-token\" (UniqueName: \"kubernetes.io/projected/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-token\") pod \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\" (UID: \"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8\") " Oct 30 09:29:32.090118 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.090047 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istiod-ca-cert" pod "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" (UID: "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:32.090508 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.090483 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-credential-socket" pod "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" (UID: "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:32.091669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.091319 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-data" pod "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" (UID: "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:32.092070 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.092048 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-workload-socket" pod "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" (UID: "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:32.092661 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.092641 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.092817 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.092803 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.092921 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.092907 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.093017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.093005 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.093118 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.093097 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.098025 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.097996 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-envoy" pod "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" (UID: "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:32.098140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.098081 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-token" pod "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" (UID: "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:32.098301 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.098275 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-kube-api-access-s576l" pod "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" (UID: "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8"). InnerVolumeSpecName "kube-api-access-s576l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:32.098411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.098313 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-podinfo" pod "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" (UID: "13923ad4-0e7f-44f5-8eb3-cd84b0d044c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:29:32.194570 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.194536 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.194570 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.194568 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.194570 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.194578 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s576l\" (UniqueName: \"kubernetes.io/projected/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-kube-api-access-s576l\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.194838 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.194604 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.287410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.287376 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj" Oct 30 09:29:32.287410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.287394 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj" event={"ID":"ddd86d9a-04a2-40f3-b74c-2b92781a0b59","Type":"ContainerDied","Data":"e2005708f6fa338be39829da4876f7a21c7bb63dfbc5d64ea888e5952b2e2e9e"} Oct 30 09:29:32.287672 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.287477 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2005708f6fa338be39829da4876f7a21c7bb63dfbc5d64ea888e5952b2e2e9e" Oct 30 09:29:32.289293 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.289266 5938 generic.go:356] "Generic (PLEG): container finished" podID="13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" containerID="8e40ec740a8035e020b1fd43aca281322226955dcbb2cf9e3bf125d27c7485e1" exitCode=0 Oct 30 09:29:32.289431 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.289337 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" Oct 30 09:29:32.289431 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.289420 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" event={"ID":"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8","Type":"ContainerDied","Data":"8e40ec740a8035e020b1fd43aca281322226955dcbb2cf9e3bf125d27c7485e1"} Oct 30 09:29:32.289534 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.289464 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr" event={"ID":"13923ad4-0e7f-44f5-8eb3-cd84b0d044c8","Type":"ContainerDied","Data":"c2cc2b01ea135bddd80bd9a7f418dd9385360d26cd5483e3a8d73cebffe27e79"} Oct 30 09:29:32.289534 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.289494 5938 scope.go:122] "RemoveContainer" containerID="8e40ec740a8035e020b1fd43aca281322226955dcbb2cf9e3bf125d27c7485e1" Oct 30 09:29:32.317626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.314180 5938 scope.go:122] "RemoveContainer" containerID="8e40ec740a8035e020b1fd43aca281322226955dcbb2cf9e3bf125d27c7485e1" Oct 30 09:29:32.317626 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:29:32.314565 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e40ec740a8035e020b1fd43aca281322226955dcbb2cf9e3bf125d27c7485e1\": container with ID starting with 8e40ec740a8035e020b1fd43aca281322226955dcbb2cf9e3bf125d27c7485e1 not found: ID does not exist" containerID="8e40ec740a8035e020b1fd43aca281322226955dcbb2cf9e3bf125d27c7485e1" Oct 30 09:29:32.317626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.314612 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e40ec740a8035e020b1fd43aca281322226955dcbb2cf9e3bf125d27c7485e1"} err="failed to get container status \"8e40ec740a8035e020b1fd43aca281322226955dcbb2cf9e3bf125d27c7485e1\": rpc error: code = NotFound desc = could not find container \"8e40ec740a8035e020b1fd43aca281322226955dcbb2cf9e3bf125d27c7485e1\": container with ID starting with 8e40ec740a8035e020b1fd43aca281322226955dcbb2cf9e3bf125d27c7485e1 not found: ID does not exist" Oct 30 09:29:32.350086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.350002 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr"] Oct 30 09:29:32.363473 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.363440 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tfjjr"] Oct 30 09:29:32.538858 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.538819 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz"] Oct 30 09:29:32.539408 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.539391 5938 cpu_manager.go:414] "containerMap: removing container" podUID="13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" containerName="istio-proxy" Oct 30 09:29:32.539483 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.539409 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" containerName="istio-proxy" Oct 30 09:29:32.539483 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.539432 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ddd86d9a-04a2-40f3-b74c-2b92781a0b59" containerName="revoke-keys" Oct 30 09:29:32.539483 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.539437 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd86d9a-04a2-40f3-b74c-2b92781a0b59" containerName="revoke-keys" Oct 30 09:29:32.539680 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.539538 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" containerName="istio-proxy" Oct 30 09:29:32.539680 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.539549 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd86d9a-04a2-40f3-b74c-2b92781a0b59" containerName="revoke-keys" Oct 30 09:29:32.543176 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.543158 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz" Oct 30 09:29:32.561731 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.561701 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz"] Oct 30 09:29:32.562388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.562364 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:29:32.562542 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.562522 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:29:32.600863 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.600755 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0031b941-1629-4bc1-81b6-c095ae05d6f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8db5c9dd-bhzfz\" (UID: \"0031b941-1629-4bc1-81b6-c095ae05d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz" Oct 30 09:29:32.601029 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.600875 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0031b941-1629-4bc1-81b6-c095ae05d6f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8db5c9dd-bhzfz\" (UID: \"0031b941-1629-4bc1-81b6-c095ae05d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz" Oct 30 09:29:32.601074 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.601058 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzz5g\" (UniqueName: \"kubernetes.io/projected/0031b941-1629-4bc1-81b6-c095ae05d6f5-kube-api-access-bzz5g\") pod \"maas-api-revoke-sub-8db5c9dd-bhzfz\" (UID: \"0031b941-1629-4bc1-81b6-c095ae05d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz" Oct 30 09:29:32.702227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.702183 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzz5g\" (UniqueName: \"kubernetes.io/projected/0031b941-1629-4bc1-81b6-c095ae05d6f5-kube-api-access-bzz5g\") pod \"maas-api-revoke-sub-8db5c9dd-bhzfz\" (UID: \"0031b941-1629-4bc1-81b6-c095ae05d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz" Oct 30 09:29:32.702455 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.702305 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0031b941-1629-4bc1-81b6-c095ae05d6f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8db5c9dd-bhzfz\" (UID: \"0031b941-1629-4bc1-81b6-c095ae05d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz" Oct 30 09:29:32.702455 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.702341 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0031b941-1629-4bc1-81b6-c095ae05d6f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8db5c9dd-bhzfz\" (UID: \"0031b941-1629-4bc1-81b6-c095ae05d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz" Oct 30 09:29:32.703001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.702980 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0031b941-1629-4bc1-81b6-c095ae05d6f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8db5c9dd-bhzfz\" (UID: \"0031b941-1629-4bc1-81b6-c095ae05d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz" Oct 30 09:29:32.705081 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.705061 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0031b941-1629-4bc1-81b6-c095ae05d6f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8db5c9dd-bhzfz\" (UID: \"0031b941-1629-4bc1-81b6-c095ae05d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz" Oct 30 09:29:32.723649 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.723580 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzz5g\" (UniqueName: \"kubernetes.io/projected/0031b941-1629-4bc1-81b6-c095ae05d6f5-kube-api-access-bzz5g\") pod \"maas-api-revoke-sub-8db5c9dd-bhzfz\" (UID: \"0031b941-1629-4bc1-81b6-c095ae05d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz" Oct 30 09:29:32.857351 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:32.857254 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz" Oct 30 09:29:33.158704 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:33.158648 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz"] Oct 30 09:29:33.161210 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:29:33.161184 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0031b941_1629_4bc1_81b6_c095ae05d6f5.slice/crio-fa5b2182c187532181e6af0a4aab626f1de97243d3cf27a67a0284502910e457 WatchSource:0}: Error finding container fa5b2182c187532181e6af0a4aab626f1de97243d3cf27a67a0284502910e457: Status 404 returned error can't find the container with id fa5b2182c187532181e6af0a4aab626f1de97243d3cf27a67a0284502910e457 Oct 30 09:29:33.297183 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:33.297141 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz" event={"ID":"0031b941-1629-4bc1-81b6-c095ae05d6f5","Type":"ContainerStarted","Data":"fa5b2182c187532181e6af0a4aab626f1de97243d3cf27a67a0284502910e457"} Oct 30 09:29:34.256967 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:34.256933 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13923ad4-0e7f-44f5-8eb3-cd84b0d044c8" path="/var/lib/kubelet/pods/13923ad4-0e7f-44f5-8eb3-cd84b0d044c8/volumes" Oct 30 09:29:34.306731 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:34.306636 5938 generic.go:356] "Generic (PLEG): container finished" podID="0031b941-1629-4bc1-81b6-c095ae05d6f5" containerID="d42ef0e0251601c89a8993c4a631c0d4c4dbbb0bab40322a679dc0d3b75fa0e3" exitCode=0 Oct 30 09:29:34.306731 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:34.306691 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz" event={"ID":"0031b941-1629-4bc1-81b6-c095ae05d6f5","Type":"ContainerDied","Data":"d42ef0e0251601c89a8993c4a631c0d4c4dbbb0bab40322a679dc0d3b75fa0e3"} Oct 30 09:29:35.477520 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:35.477493 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz" Oct 30 09:29:35.540087 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:35.540056 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0031b941-1629-4bc1-81b6-c095ae05d6f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0031b941-1629-4bc1-81b6-c095ae05d6f5-maas-api-openshift-ca\") pod \"0031b941-1629-4bc1-81b6-c095ae05d6f5\" (UID: \"0031b941-1629-4bc1-81b6-c095ae05d6f5\") " Oct 30 09:29:35.540294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:35.540184 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0031b941-1629-4bc1-81b6-c095ae05d6f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0031b941-1629-4bc1-81b6-c095ae05d6f5-maas-api-xks-ca\") pod \"0031b941-1629-4bc1-81b6-c095ae05d6f5\" (UID: \"0031b941-1629-4bc1-81b6-c095ae05d6f5\") " Oct 30 09:29:35.540294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:35.540229 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0031b941-1629-4bc1-81b6-c095ae05d6f5-kube-api-access-bzz5g\" (UniqueName: \"kubernetes.io/projected/0031b941-1629-4bc1-81b6-c095ae05d6f5-kube-api-access-bzz5g\") pod \"0031b941-1629-4bc1-81b6-c095ae05d6f5\" (UID: \"0031b941-1629-4bc1-81b6-c095ae05d6f5\") " Oct 30 09:29:35.540515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:35.540487 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0031b941-1629-4bc1-81b6-c095ae05d6f5-maas-api-openshift-ca" pod "0031b941-1629-4bc1-81b6-c095ae05d6f5" (UID: "0031b941-1629-4bc1-81b6-c095ae05d6f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:35.540851 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:35.540828 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0031b941-1629-4bc1-81b6-c095ae05d6f5-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:35.543105 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:35.543084 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0031b941-1629-4bc1-81b6-c095ae05d6f5-maas-api-xks-ca" pod "0031b941-1629-4bc1-81b6-c095ae05d6f5" (UID: "0031b941-1629-4bc1-81b6-c095ae05d6f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:35.545753 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:35.545723 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0031b941-1629-4bc1-81b6-c095ae05d6f5-kube-api-access-bzz5g" pod "0031b941-1629-4bc1-81b6-c095ae05d6f5" (UID: "0031b941-1629-4bc1-81b6-c095ae05d6f5"). InnerVolumeSpecName "kube-api-access-bzz5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:35.642245 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:35.642213 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0031b941-1629-4bc1-81b6-c095ae05d6f5-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:35.642245 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:35.642245 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzz5g\" (UniqueName: \"kubernetes.io/projected/0031b941-1629-4bc1-81b6-c095ae05d6f5-kube-api-access-bzz5g\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:36.321189 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:36.321156 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz" Oct 30 09:29:36.321393 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:36.321185 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz" event={"ID":"0031b941-1629-4bc1-81b6-c095ae05d6f5","Type":"ContainerDied","Data":"fa5b2182c187532181e6af0a4aab626f1de97243d3cf27a67a0284502910e457"} Oct 30 09:29:36.321393 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:36.321247 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa5b2182c187532181e6af0a4aab626f1de97243d3cf27a67a0284502910e457" Oct 30 09:29:41.474529 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.474486 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh"] Oct 30 09:29:41.475095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.475080 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0031b941-1629-4bc1-81b6-c095ae05d6f5" containerName="revoke-keys" Oct 30 09:29:41.475095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.475096 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0031b941-1629-4bc1-81b6-c095ae05d6f5" containerName="revoke-keys" Oct 30 09:29:41.475214 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.475187 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0031b941-1629-4bc1-81b6-c095ae05d6f5" containerName="revoke-keys" Oct 30 09:29:41.478713 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.478690 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh" Oct 30 09:29:41.497457 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.497423 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh"] Oct 30 09:29:41.507894 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.507861 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:29:41.508076 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.507940 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:29:41.628602 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.628557 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef64d976-b243-4850-9a85-de7e9483bdbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41a196ec-dgkqh\" (UID: \"ef64d976-b243-4850-9a85-de7e9483bdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh" Oct 30 09:29:41.628820 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.628717 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvlt\" (UniqueName: \"kubernetes.io/projected/ef64d976-b243-4850-9a85-de7e9483bdbf-kube-api-access-5lvlt\") pod \"maas-api-revoke-sub-41a196ec-dgkqh\" (UID: \"ef64d976-b243-4850-9a85-de7e9483bdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh" Oct 30 09:29:41.628820 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.628807 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef64d976-b243-4850-9a85-de7e9483bdbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41a196ec-dgkqh\" (UID: \"ef64d976-b243-4850-9a85-de7e9483bdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh" Oct 30 09:29:41.730363 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.730269 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef64d976-b243-4850-9a85-de7e9483bdbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41a196ec-dgkqh\" (UID: \"ef64d976-b243-4850-9a85-de7e9483bdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh" Oct 30 09:29:41.730363 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.730337 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvlt\" (UniqueName: \"kubernetes.io/projected/ef64d976-b243-4850-9a85-de7e9483bdbf-kube-api-access-5lvlt\") pod \"maas-api-revoke-sub-41a196ec-dgkqh\" (UID: \"ef64d976-b243-4850-9a85-de7e9483bdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh" Oct 30 09:29:41.730664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.730421 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef64d976-b243-4850-9a85-de7e9483bdbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41a196ec-dgkqh\" (UID: \"ef64d976-b243-4850-9a85-de7e9483bdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh" Oct 30 09:29:41.731006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.730981 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef64d976-b243-4850-9a85-de7e9483bdbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41a196ec-dgkqh\" (UID: \"ef64d976-b243-4850-9a85-de7e9483bdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh" Oct 30 09:29:41.733258 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.733219 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef64d976-b243-4850-9a85-de7e9483bdbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41a196ec-dgkqh\" (UID: \"ef64d976-b243-4850-9a85-de7e9483bdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh" Oct 30 09:29:41.751909 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.751876 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvlt\" (UniqueName: \"kubernetes.io/projected/ef64d976-b243-4850-9a85-de7e9483bdbf-kube-api-access-5lvlt\") pod \"maas-api-revoke-sub-41a196ec-dgkqh\" (UID: \"ef64d976-b243-4850-9a85-de7e9483bdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh" Oct 30 09:29:41.792986 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:41.792948 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh" Oct 30 09:29:42.102423 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:29:42.102375 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef64d976_b243_4850_9a85_de7e9483bdbf.slice/crio-947e7632edabe5158463908e5931db68d4e2174aeb200b373faebb9603183242 WatchSource:0}: Error finding container 947e7632edabe5158463908e5931db68d4e2174aeb200b373faebb9603183242: Status 404 returned error can't find the container with id 947e7632edabe5158463908e5931db68d4e2174aeb200b373faebb9603183242 Oct 30 09:29:42.107509 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:42.105986 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh"] Oct 30 09:29:42.369329 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:42.369289 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh" event={"ID":"ef64d976-b243-4850-9a85-de7e9483bdbf","Type":"ContainerStarted","Data":"947e7632edabe5158463908e5931db68d4e2174aeb200b373faebb9603183242"} Oct 30 09:29:43.377746 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:43.377713 5938 generic.go:356] "Generic (PLEG): container finished" podID="ef64d976-b243-4850-9a85-de7e9483bdbf" containerID="2c9ee1217a23799611dbaf0aed2a01b3064fd3c5cd1e343a1deea890fb82198f" exitCode=0 Oct 30 09:29:43.378139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:43.377769 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh" event={"ID":"ef64d976-b243-4850-9a85-de7e9483bdbf","Type":"ContainerDied","Data":"2c9ee1217a23799611dbaf0aed2a01b3064fd3c5cd1e343a1deea890fb82198f"} Oct 30 09:29:44.561367 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:44.561337 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh" Oct 30 09:29:44.573431 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:44.573397 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8"] Oct 30 09:29:44.587077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:44.587044 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51e091-9rvq8"] Oct 30 09:29:44.625859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:44.625828 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef64d976-b243-4850-9a85-de7e9483bdbf-kube-api-access-5lvlt\" (UniqueName: \"kubernetes.io/projected/ef64d976-b243-4850-9a85-de7e9483bdbf-kube-api-access-5lvlt\") pod \"ef64d976-b243-4850-9a85-de7e9483bdbf\" (UID: \"ef64d976-b243-4850-9a85-de7e9483bdbf\") " Oct 30 09:29:44.626061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:44.625970 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef64d976-b243-4850-9a85-de7e9483bdbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef64d976-b243-4850-9a85-de7e9483bdbf-maas-api-xks-ca\") pod \"ef64d976-b243-4850-9a85-de7e9483bdbf\" (UID: \"ef64d976-b243-4850-9a85-de7e9483bdbf\") " Oct 30 09:29:44.626141 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:44.626120 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef64d976-b243-4850-9a85-de7e9483bdbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef64d976-b243-4850-9a85-de7e9483bdbf-maas-api-openshift-ca\") pod \"ef64d976-b243-4850-9a85-de7e9483bdbf\" (UID: \"ef64d976-b243-4850-9a85-de7e9483bdbf\") " Oct 30 09:29:44.626603 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:44.626525 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef64d976-b243-4850-9a85-de7e9483bdbf-maas-api-openshift-ca" pod "ef64d976-b243-4850-9a85-de7e9483bdbf" (UID: "ef64d976-b243-4850-9a85-de7e9483bdbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:44.626831 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:44.626673 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef64d976-b243-4850-9a85-de7e9483bdbf-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:44.628294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:44.628264 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef64d976-b243-4850-9a85-de7e9483bdbf-kube-api-access-5lvlt" pod "ef64d976-b243-4850-9a85-de7e9483bdbf" (UID: "ef64d976-b243-4850-9a85-de7e9483bdbf"). InnerVolumeSpecName "kube-api-access-5lvlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:44.628812 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:44.628787 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef64d976-b243-4850-9a85-de7e9483bdbf-maas-api-xks-ca" pod "ef64d976-b243-4850-9a85-de7e9483bdbf" (UID: "ef64d976-b243-4850-9a85-de7e9483bdbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:44.728424 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:44.728377 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lvlt\" (UniqueName: \"kubernetes.io/projected/ef64d976-b243-4850-9a85-de7e9483bdbf-kube-api-access-5lvlt\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:44.728424 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:44.728416 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef64d976-b243-4850-9a85-de7e9483bdbf-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:45.410383 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:45.410347 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh" Oct 30 09:29:45.410577 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:45.410380 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh" event={"ID":"ef64d976-b243-4850-9a85-de7e9483bdbf","Type":"ContainerDied","Data":"947e7632edabe5158463908e5931db68d4e2174aeb200b373faebb9603183242"} Oct 30 09:29:45.410577 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:45.410433 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="947e7632edabe5158463908e5931db68d4e2174aeb200b373faebb9603183242" Oct 30 09:29:45.883389 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:29:45.883343 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:29:46.190655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.190553 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92"] Oct 30 09:29:46.191127 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.191112 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ef64d976-b243-4850-9a85-de7e9483bdbf" containerName="revoke-keys" Oct 30 09:29:46.191180 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.191129 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef64d976-b243-4850-9a85-de7e9483bdbf" containerName="revoke-keys" Oct 30 09:29:46.191242 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.191232 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef64d976-b243-4850-9a85-de7e9483bdbf" containerName="revoke-keys" Oct 30 09:29:46.194900 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.194876 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" Oct 30 09:29:46.213754 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.213714 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92"] Oct 30 09:29:46.224331 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.224299 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:29:46.224490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.224365 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:29:46.247763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.247723 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c4fbd3-24cb-46f6-ad01-962114ae7ece-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4cf242f-4dw92\" (UID: \"00c4fbd3-24cb-46f6-ad01-962114ae7ece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" Oct 30 09:29:46.247987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.247814 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c4fbd3-24cb-46f6-ad01-962114ae7ece-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4cf242f-4dw92\" (UID: \"00c4fbd3-24cb-46f6-ad01-962114ae7ece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" Oct 30 09:29:46.247987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.247947 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8tfw\" (UniqueName: \"kubernetes.io/projected/00c4fbd3-24cb-46f6-ad01-962114ae7ece-kube-api-access-j8tfw\") pod \"maas-api-revoke-sub-e4cf242f-4dw92\" (UID: \"00c4fbd3-24cb-46f6-ad01-962114ae7ece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" Oct 30 09:29:46.257221 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.257185 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3f0797f-5874-4771-9b53-173dcf49b89f" path="/var/lib/kubelet/pods/b3f0797f-5874-4771-9b53-173dcf49b89f/volumes" Oct 30 09:29:46.349442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.349399 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c4fbd3-24cb-46f6-ad01-962114ae7ece-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4cf242f-4dw92\" (UID: \"00c4fbd3-24cb-46f6-ad01-962114ae7ece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" Oct 30 09:29:46.349688 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.349461 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c4fbd3-24cb-46f6-ad01-962114ae7ece-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4cf242f-4dw92\" (UID: \"00c4fbd3-24cb-46f6-ad01-962114ae7ece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" Oct 30 09:29:46.349688 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.349645 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8tfw\" (UniqueName: \"kubernetes.io/projected/00c4fbd3-24cb-46f6-ad01-962114ae7ece-kube-api-access-j8tfw\") pod \"maas-api-revoke-sub-e4cf242f-4dw92\" (UID: \"00c4fbd3-24cb-46f6-ad01-962114ae7ece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" Oct 30 09:29:46.350182 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.350161 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c4fbd3-24cb-46f6-ad01-962114ae7ece-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4cf242f-4dw92\" (UID: \"00c4fbd3-24cb-46f6-ad01-962114ae7ece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" Oct 30 09:29:46.352330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.352299 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c4fbd3-24cb-46f6-ad01-962114ae7ece-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4cf242f-4dw92\" (UID: \"00c4fbd3-24cb-46f6-ad01-962114ae7ece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" Oct 30 09:29:46.370889 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.370857 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8tfw\" (UniqueName: \"kubernetes.io/projected/00c4fbd3-24cb-46f6-ad01-962114ae7ece-kube-api-access-j8tfw\") pod \"maas-api-revoke-sub-e4cf242f-4dw92\" (UID: \"00c4fbd3-24cb-46f6-ad01-962114ae7ece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" Oct 30 09:29:46.509866 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.509775 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" Oct 30 09:29:46.816385 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:46.816357 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92"] Oct 30 09:29:46.818216 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:29:46.818183 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c4fbd3_24cb_46f6_ad01_962114ae7ece.slice/crio-75ca7335e693539250290b0f1cb11a84d77000a9b7e986ae4bebbe6bda07fd1e WatchSource:0}: Error finding container 75ca7335e693539250290b0f1cb11a84d77000a9b7e986ae4bebbe6bda07fd1e: Status 404 returned error can't find the container with id 75ca7335e693539250290b0f1cb11a84d77000a9b7e986ae4bebbe6bda07fd1e Oct 30 09:29:47.439070 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:47.438953 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" event={"ID":"00c4fbd3-24cb-46f6-ad01-962114ae7ece","Type":"ContainerStarted","Data":"c2152deb65a560a517e9910e623c7263bf6adf05101cbcb2ad2071dc1ab49efc"} Oct 30 09:29:47.439507 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:47.439069 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" event={"ID":"00c4fbd3-24cb-46f6-ad01-962114ae7ece","Type":"ContainerStarted","Data":"75ca7335e693539250290b0f1cb11a84d77000a9b7e986ae4bebbe6bda07fd1e"} Oct 30 09:29:47.480801 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:47.480748 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" podStartSLOduration=1.4807337409999999 podStartE2EDuration="1.480733741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:47.479673917 +0000 UTC m=+2105.342379738" watchObservedRunningTime="2026-10-30 09:29:47.480733741 +0000 UTC m=+2105.343439561" Oct 30 09:29:48.447866 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:48.447827 5938 generic.go:356] "Generic (PLEG): container finished" podID="00c4fbd3-24cb-46f6-ad01-962114ae7ece" containerID="c2152deb65a560a517e9910e623c7263bf6adf05101cbcb2ad2071dc1ab49efc" exitCode=0 Oct 30 09:29:48.448266 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:48.447918 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" event={"ID":"00c4fbd3-24cb-46f6-ad01-962114ae7ece","Type":"ContainerDied","Data":"c2152deb65a560a517e9910e623c7263bf6adf05101cbcb2ad2071dc1ab49efc"} Oct 30 09:29:49.628645 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.628604 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z"] Oct 30 09:29:49.634708 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.634625 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" Oct 30 09:29:49.634912 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.634753 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.654882 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.654846 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z"] Oct 30 09:29:49.692439 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.692394 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c4fbd3-24cb-46f6-ad01-962114ae7ece-kube-api-access-j8tfw\" (UniqueName: \"kubernetes.io/projected/00c4fbd3-24cb-46f6-ad01-962114ae7ece-kube-api-access-j8tfw\") pod \"00c4fbd3-24cb-46f6-ad01-962114ae7ece\" (UID: \"00c4fbd3-24cb-46f6-ad01-962114ae7ece\") " Oct 30 09:29:49.692809 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.692455 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c4fbd3-24cb-46f6-ad01-962114ae7ece-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c4fbd3-24cb-46f6-ad01-962114ae7ece-maas-api-openshift-ca\") pod \"00c4fbd3-24cb-46f6-ad01-962114ae7ece\" (UID: \"00c4fbd3-24cb-46f6-ad01-962114ae7ece\") " Oct 30 09:29:49.692992 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.692962 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00c4fbd3-24cb-46f6-ad01-962114ae7ece-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c4fbd3-24cb-46f6-ad01-962114ae7ece-maas-api-xks-ca\") pod \"00c4fbd3-24cb-46f6-ad01-962114ae7ece\" (UID: \"00c4fbd3-24cb-46f6-ad01-962114ae7ece\") " Oct 30 09:29:49.693166 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.693144 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c4fbd3-24cb-46f6-ad01-962114ae7ece-maas-api-openshift-ca" pod "00c4fbd3-24cb-46f6-ad01-962114ae7ece" (UID: "00c4fbd3-24cb-46f6-ad01-962114ae7ece"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:49.693304 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.693276 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.693433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.693335 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.693433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.693433 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.693691 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.693466 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.693691 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.693620 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqgdr\" (UniqueName: \"kubernetes.io/projected/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-kube-api-access-jqgdr\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.693818 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.693677 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.693818 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.693727 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.693929 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.693886 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.693929 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.693913 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.694046 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.694029 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c4fbd3-24cb-46f6-ad01-962114ae7ece-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:49.695315 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.695283 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c4fbd3-24cb-46f6-ad01-962114ae7ece-kube-api-access-j8tfw" pod "00c4fbd3-24cb-46f6-ad01-962114ae7ece" (UID: "00c4fbd3-24cb-46f6-ad01-962114ae7ece"). InnerVolumeSpecName "kube-api-access-j8tfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:49.695928 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.695898 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c4fbd3-24cb-46f6-ad01-962114ae7ece-maas-api-xks-ca" pod "00c4fbd3-24cb-46f6-ad01-962114ae7ece" (UID: "00c4fbd3-24cb-46f6-ad01-962114ae7ece"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:49.795091 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.794996 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqgdr\" (UniqueName: \"kubernetes.io/projected/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-kube-api-access-jqgdr\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.795091 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.795062 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.795279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.795097 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.795279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.795193 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.795279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.795222 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.795402 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.795315 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.795402 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.795352 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.795402 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.795376 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.795512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.795415 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.795512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.795477 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.795512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.795493 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.795707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.795687 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c4fbd3-24cb-46f6-ad01-962114ae7ece-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:49.795767 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.795715 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8tfw\" (UniqueName: \"kubernetes.io/projected/00c4fbd3-24cb-46f6-ad01-962114ae7ece-kube-api-access-j8tfw\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:49.795832 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.795816 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.795914 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.795894 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.796402 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.796369 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.797961 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.797940 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.798294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.798271 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.816773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.816738 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.816978 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.816954 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqgdr\" (UniqueName: \"kubernetes.io/projected/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-kube-api-access-jqgdr\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:49.955349 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:49.955311 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:50.291075 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:50.291042 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z"] Oct 30 09:29:50.292297 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:29:50.292266 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c15e50_b4b3_4b40_92a2_a0f392f2e63e.slice/crio-0485bb271c58ec8ad09971cca5a7cee8fb3ba3fc286aa86527268fb1424d624e WatchSource:0}: Error finding container 0485bb271c58ec8ad09971cca5a7cee8fb3ba3fc286aa86527268fb1424d624e: Status 404 returned error can't find the container with id 0485bb271c58ec8ad09971cca5a7cee8fb3ba3fc286aa86527268fb1424d624e Oct 30 09:29:50.295182 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:50.295137 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:29:50.295286 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:50.295222 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:29:50.295286 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:50.295256 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:29:50.466899 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:50.466867 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" Oct 30 09:29:50.467100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:50.466889 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92" event={"ID":"00c4fbd3-24cb-46f6-ad01-962114ae7ece","Type":"ContainerDied","Data":"75ca7335e693539250290b0f1cb11a84d77000a9b7e986ae4bebbe6bda07fd1e"} Oct 30 09:29:50.467100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:50.466961 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75ca7335e693539250290b0f1cb11a84d77000a9b7e986ae4bebbe6bda07fd1e" Oct 30 09:29:50.469255 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:50.469222 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" event={"ID":"28c15e50-b4b3-4b40-92a2-a0f392f2e63e","Type":"ContainerStarted","Data":"b8a4749a111d2d17fd442bec7e4b200d1534abcc4849e4a4049aeaf98b486157"} Oct 30 09:29:50.469451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:50.469287 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" event={"ID":"28c15e50-b4b3-4b40-92a2-a0f392f2e63e","Type":"ContainerStarted","Data":"0485bb271c58ec8ad09971cca5a7cee8fb3ba3fc286aa86527268fb1424d624e"} Oct 30 09:29:50.516975 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:50.516843 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" podStartSLOduration=1.516822698 podStartE2EDuration="1.516822698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:50.515117532 +0000 UTC m=+2108.377823352" watchObservedRunningTime="2026-10-30 09:29:50.516822698 +0000 UTC m=+2108.379528519" Oct 30 09:29:50.956330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:50.956290 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:51.961613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:51.960848 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 09:29:51.961613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:51.960921 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" podUID="28c15e50-b4b3-4b40-92a2-a0f392f2e63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 09:29:52.852364 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:52.852325 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb"] Oct 30 09:29:52.852617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:52.852559 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" podUID="6e0e5389-b8e8-4033-a7b3-98830bf147af" containerName="istio-proxy" containerID="cri-o://a96e59da7fa89e0012a02cd670ab39b4f32a1db680a18408bfaeee06308630bc" gracePeriod=30 Oct 30 09:29:52.956321 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:52.956281 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 09:29:52.956514 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:52.956347 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" podUID="28c15e50-b4b3-4b40-92a2-a0f392f2e63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 09:29:53.956024 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:53.955989 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 09:29:53.956459 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:53.956042 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" podUID="28c15e50-b4b3-4b40-92a2-a0f392f2e63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 09:29:54.586376 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.586335 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq"] Oct 30 09:29:54.587229 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.587206 5938 cpu_manager.go:414] "containerMap: removing container" podUID="00c4fbd3-24cb-46f6-ad01-962114ae7ece" containerName="revoke-keys" Oct 30 09:29:54.587304 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.587234 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c4fbd3-24cb-46f6-ad01-962114ae7ece" containerName="revoke-keys" Oct 30 09:29:54.587397 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.587385 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c4fbd3-24cb-46f6-ad01-962114ae7ece" containerName="revoke-keys" Oct 30 09:29:54.591846 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.591814 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" Oct 30 09:29:54.611319 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.611288 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq"] Oct 30 09:29:54.628616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.624880 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:29:54.628616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.625023 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:29:54.749619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.746906 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/653b44ab-fdf8-4f1a-bd3c-946173e53afe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bf585ce-vf8zq\" (UID: \"653b44ab-fdf8-4f1a-bd3c-946173e53afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" Oct 30 09:29:54.749619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.747004 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlzxw\" (UniqueName: \"kubernetes.io/projected/653b44ab-fdf8-4f1a-bd3c-946173e53afe-kube-api-access-zlzxw\") pod \"maas-api-revoke-sub-9bf585ce-vf8zq\" (UID: \"653b44ab-fdf8-4f1a-bd3c-946173e53afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" Oct 30 09:29:54.749619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.747055 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/653b44ab-fdf8-4f1a-bd3c-946173e53afe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bf585ce-vf8zq\" (UID: \"653b44ab-fdf8-4f1a-bd3c-946173e53afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" Oct 30 09:29:54.851124 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.850603 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/653b44ab-fdf8-4f1a-bd3c-946173e53afe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bf585ce-vf8zq\" (UID: \"653b44ab-fdf8-4f1a-bd3c-946173e53afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" Oct 30 09:29:54.851124 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.850696 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzxw\" (UniqueName: \"kubernetes.io/projected/653b44ab-fdf8-4f1a-bd3c-946173e53afe-kube-api-access-zlzxw\") pod \"maas-api-revoke-sub-9bf585ce-vf8zq\" (UID: \"653b44ab-fdf8-4f1a-bd3c-946173e53afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" Oct 30 09:29:54.851124 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.850746 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/653b44ab-fdf8-4f1a-bd3c-946173e53afe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bf585ce-vf8zq\" (UID: \"653b44ab-fdf8-4f1a-bd3c-946173e53afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" Oct 30 09:29:54.851668 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.851620 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/653b44ab-fdf8-4f1a-bd3c-946173e53afe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bf585ce-vf8zq\" (UID: \"653b44ab-fdf8-4f1a-bd3c-946173e53afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" Oct 30 09:29:54.855079 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.855047 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/653b44ab-fdf8-4f1a-bd3c-946173e53afe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bf585ce-vf8zq\" (UID: \"653b44ab-fdf8-4f1a-bd3c-946173e53afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" Oct 30 09:29:54.873212 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.873161 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlzxw\" (UniqueName: \"kubernetes.io/projected/653b44ab-fdf8-4f1a-bd3c-946173e53afe-kube-api-access-zlzxw\") pod \"maas-api-revoke-sub-9bf585ce-vf8zq\" (UID: \"653b44ab-fdf8-4f1a-bd3c-946173e53afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" Oct 30 09:29:54.909481 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.909431 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" Oct 30 09:29:54.958183 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.957248 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 09:29:54.958183 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:54.957309 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" podUID="28c15e50-b4b3-4b40-92a2-a0f392f2e63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 09:29:55.314953 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:55.314921 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq"] Oct 30 09:29:55.315204 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:29:55.315181 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653b44ab_fdf8_4f1a_bd3c_946173e53afe.slice/crio-96b78d0d0a746705499c841238f4c1a3bd39b3683e900d3c78b1291d22868fc6 WatchSource:0}: Error finding container 96b78d0d0a746705499c841238f4c1a3bd39b3683e900d3c78b1291d22868fc6: Status 404 returned error can't find the container with id 96b78d0d0a746705499c841238f4c1a3bd39b3683e900d3c78b1291d22868fc6 Oct 30 09:29:55.507071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:55.507034 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" event={"ID":"653b44ab-fdf8-4f1a-bd3c-946173e53afe","Type":"ContainerStarted","Data":"96b78d0d0a746705499c841238f4c1a3bd39b3683e900d3c78b1291d22868fc6"} Oct 30 09:29:55.956996 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:55.956956 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 09:29:55.957306 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:55.957267 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" podUID="28c15e50-b4b3-4b40-92a2-a0f392f2e63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 09:29:56.574356 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:56.574259 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" event={"ID":"653b44ab-fdf8-4f1a-bd3c-946173e53afe","Type":"ContainerStarted","Data":"3214b73aa64b7227c1e342d43e6116701bdbc1bc83f475c4b6a73dc6fc502e66"} Oct 30 09:29:56.618239 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:56.618167 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" podStartSLOduration=2.61814993 podStartE2EDuration="2.61814993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:56.6167592 +0000 UTC m=+2114.479465021" watchObservedRunningTime="2026-10-30 09:29:56.61814993 +0000 UTC m=+2114.480855754" Oct 30 09:29:56.958109 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:56.957888 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 09:29:56.959655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:56.959550 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" podUID="28c15e50-b4b3-4b40-92a2-a0f392f2e63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 09:29:57.584117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:57.584082 5938 generic.go:356] "Generic (PLEG): container finished" podID="653b44ab-fdf8-4f1a-bd3c-946173e53afe" containerID="3214b73aa64b7227c1e342d43e6116701bdbc1bc83f475c4b6a73dc6fc502e66" exitCode=0 Oct 30 09:29:57.584637 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:57.584153 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" event={"ID":"653b44ab-fdf8-4f1a-bd3c-946173e53afe","Type":"ContainerDied","Data":"3214b73aa64b7227c1e342d43e6116701bdbc1bc83f475c4b6a73dc6fc502e66"} Oct 30 09:29:58.313923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.313598 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:29:58.379304 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.379273 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:58.379838 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.379811 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:58.383910 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.383877 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:29:58.416572 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.416481 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-workload-socket\") pod \"6e0e5389-b8e8-4033-a7b3-98830bf147af\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " Oct 30 09:29:58.416903 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.416872 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-workload-socket" pod "6e0e5389-b8e8-4033-a7b3-98830bf147af" (UID: "6e0e5389-b8e8-4033-a7b3-98830bf147af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:58.417094 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.417075 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-credential-socket\") pod \"6e0e5389-b8e8-4033-a7b3-98830bf147af\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " Oct 30 09:29:58.417309 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.417291 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e0e5389-b8e8-4033-a7b3-98830bf147af-kube-api-access-6wg7f\" (UniqueName: \"kubernetes.io/projected/6e0e5389-b8e8-4033-a7b3-98830bf147af-kube-api-access-6wg7f\") pod \"6e0e5389-b8e8-4033-a7b3-98830bf147af\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " Oct 30 09:29:58.418429 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.418406 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-envoy\") pod \"6e0e5389-b8e8-4033-a7b3-98830bf147af\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " Oct 30 09:29:58.418764 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.418750 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-data\") pod \"6e0e5389-b8e8-4033-a7b3-98830bf147af\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " Oct 30 09:29:58.419751 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.417411 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-credential-socket" pod "6e0e5389-b8e8-4033-a7b3-98830bf147af" (UID: "6e0e5389-b8e8-4033-a7b3-98830bf147af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:58.419876 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.419608 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-data" pod "6e0e5389-b8e8-4033-a7b3-98830bf147af" (UID: "6e0e5389-b8e8-4033-a7b3-98830bf147af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:58.419969 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.419950 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e0e5389-b8e8-4033-a7b3-98830bf147af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e0e5389-b8e8-4033-a7b3-98830bf147af-istiod-ca-cert\") pod \"6e0e5389-b8e8-4033-a7b3-98830bf147af\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " Oct 30 09:29:58.420167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.420140 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0e5389-b8e8-4033-a7b3-98830bf147af-istiod-ca-cert" pod "6e0e5389-b8e8-4033-a7b3-98830bf147af" (UID: "6e0e5389-b8e8-4033-a7b3-98830bf147af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:58.420274 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.420258 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-token\" (UniqueName: \"kubernetes.io/projected/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-token\") pod \"6e0e5389-b8e8-4033-a7b3-98830bf147af\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " Oct 30 09:29:58.420425 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.420412 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-podinfo\") pod \"6e0e5389-b8e8-4033-a7b3-98830bf147af\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " Oct 30 09:29:58.420918 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.420902 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-workload-certs\") pod \"6e0e5389-b8e8-4033-a7b3-98830bf147af\" (UID: \"6e0e5389-b8e8-4033-a7b3-98830bf147af\") " Oct 30 09:29:58.421375 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.421043 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0e5389-b8e8-4033-a7b3-98830bf147af-kube-api-access-6wg7f" pod "6e0e5389-b8e8-4033-a7b3-98830bf147af" (UID: "6e0e5389-b8e8-4033-a7b3-98830bf147af"). InnerVolumeSpecName "kube-api-access-6wg7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:58.421375 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.421251 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-workload-certs" pod "6e0e5389-b8e8-4033-a7b3-98830bf147af" (UID: "6e0e5389-b8e8-4033-a7b3-98830bf147af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:58.425348 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.425231 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wg7f\" (UniqueName: \"kubernetes.io/projected/6e0e5389-b8e8-4033-a7b3-98830bf147af-kube-api-access-6wg7f\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.425348 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.425264 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.425348 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.425277 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e0e5389-b8e8-4033-a7b3-98830bf147af-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.425348 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.425291 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.425348 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.425305 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.425348 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.425319 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.437405 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.437346 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-token" pod "6e0e5389-b8e8-4033-a7b3-98830bf147af" (UID: "6e0e5389-b8e8-4033-a7b3-98830bf147af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:58.439969 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.439871 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-podinfo" pod "6e0e5389-b8e8-4033-a7b3-98830bf147af" (UID: "6e0e5389-b8e8-4033-a7b3-98830bf147af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:29:58.441135 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.441084 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-envoy" pod "6e0e5389-b8e8-4033-a7b3-98830bf147af" (UID: "6e0e5389-b8e8-4033-a7b3-98830bf147af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:58.529016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.528979 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.529294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.529263 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.529450 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.529423 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e0e5389-b8e8-4033-a7b3-98830bf147af-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.672803 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.672762 5938 generic.go:356] "Generic (PLEG): container finished" podID="6e0e5389-b8e8-4033-a7b3-98830bf147af" containerID="a96e59da7fa89e0012a02cd670ab39b4f32a1db680a18408bfaeee06308630bc" exitCode=0 Oct 30 09:29:58.673345 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.673137 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" Oct 30 09:29:58.677756 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.677701 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" event={"ID":"6e0e5389-b8e8-4033-a7b3-98830bf147af","Type":"ContainerDied","Data":"a96e59da7fa89e0012a02cd670ab39b4f32a1db680a18408bfaeee06308630bc"} Oct 30 09:29:58.677921 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.677873 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb" event={"ID":"6e0e5389-b8e8-4033-a7b3-98830bf147af","Type":"ContainerDied","Data":"fbd35e038a25badc123253bbd13339d27de54c5113d40f256a984ff4cedb977b"} Oct 30 09:29:58.677980 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.677940 5938 scope.go:122] "RemoveContainer" containerID="a96e59da7fa89e0012a02cd670ab39b4f32a1db680a18408bfaeee06308630bc" Oct 30 09:29:58.701515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.701488 5938 scope.go:122] "RemoveContainer" containerID="a96e59da7fa89e0012a02cd670ab39b4f32a1db680a18408bfaeee06308630bc" Oct 30 09:29:58.702067 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:29:58.702028 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a96e59da7fa89e0012a02cd670ab39b4f32a1db680a18408bfaeee06308630bc\": container with ID starting with a96e59da7fa89e0012a02cd670ab39b4f32a1db680a18408bfaeee06308630bc not found: ID does not exist" containerID="a96e59da7fa89e0012a02cd670ab39b4f32a1db680a18408bfaeee06308630bc" Oct 30 09:29:58.702173 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.702085 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96e59da7fa89e0012a02cd670ab39b4f32a1db680a18408bfaeee06308630bc"} err="failed to get container status \"a96e59da7fa89e0012a02cd670ab39b4f32a1db680a18408bfaeee06308630bc\": rpc error: code = NotFound desc = could not find container \"a96e59da7fa89e0012a02cd670ab39b4f32a1db680a18408bfaeee06308630bc\": container with ID starting with a96e59da7fa89e0012a02cd670ab39b4f32a1db680a18408bfaeee06308630bc not found: ID does not exist" Oct 30 09:29:58.768755 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.768714 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb"] Oct 30 09:29:58.771721 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.771685 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-k6rbb"] Oct 30 09:29:58.981603 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:58.981229 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" Oct 30 09:29:59.037850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:59.037805 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653b44ab-fdf8-4f1a-bd3c-946173e53afe-kube-api-access-zlzxw\" (UniqueName: \"kubernetes.io/projected/653b44ab-fdf8-4f1a-bd3c-946173e53afe-kube-api-access-zlzxw\") pod \"653b44ab-fdf8-4f1a-bd3c-946173e53afe\" (UID: \"653b44ab-fdf8-4f1a-bd3c-946173e53afe\") " Oct 30 09:29:59.038261 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:59.038239 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/653b44ab-fdf8-4f1a-bd3c-946173e53afe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/653b44ab-fdf8-4f1a-bd3c-946173e53afe-maas-api-xks-ca\") pod \"653b44ab-fdf8-4f1a-bd3c-946173e53afe\" (UID: \"653b44ab-fdf8-4f1a-bd3c-946173e53afe\") " Oct 30 09:29:59.038414 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:59.038399 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653b44ab-fdf8-4f1a-bd3c-946173e53afe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/653b44ab-fdf8-4f1a-bd3c-946173e53afe-maas-api-openshift-ca\") pod \"653b44ab-fdf8-4f1a-bd3c-946173e53afe\" (UID: \"653b44ab-fdf8-4f1a-bd3c-946173e53afe\") " Oct 30 09:29:59.039336 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:59.039299 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653b44ab-fdf8-4f1a-bd3c-946173e53afe-maas-api-openshift-ca" pod "653b44ab-fdf8-4f1a-bd3c-946173e53afe" (UID: "653b44ab-fdf8-4f1a-bd3c-946173e53afe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:59.039657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:59.039637 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/653b44ab-fdf8-4f1a-bd3c-946173e53afe-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:59.040887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:59.040640 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653b44ab-fdf8-4f1a-bd3c-946173e53afe-kube-api-access-zlzxw" pod "653b44ab-fdf8-4f1a-bd3c-946173e53afe" (UID: "653b44ab-fdf8-4f1a-bd3c-946173e53afe"). InnerVolumeSpecName "kube-api-access-zlzxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:59.042042 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:59.042014 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/653b44ab-fdf8-4f1a-bd3c-946173e53afe-maas-api-xks-ca" pod "653b44ab-fdf8-4f1a-bd3c-946173e53afe" (UID: "653b44ab-fdf8-4f1a-bd3c-946173e53afe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:59.150505 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:59.149952 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/653b44ab-fdf8-4f1a-bd3c-946173e53afe-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:59.150505 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:59.149990 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlzxw\" (UniqueName: \"kubernetes.io/projected/653b44ab-fdf8-4f1a-bd3c-946173e53afe-kube-api-access-zlzxw\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:29:59.782182 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:59.782129 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" event={"ID":"653b44ab-fdf8-4f1a-bd3c-946173e53afe","Type":"ContainerDied","Data":"96b78d0d0a746705499c841238f4c1a3bd39b3683e900d3c78b1291d22868fc6"} Oct 30 09:29:59.782819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:59.782779 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96b78d0d0a746705499c841238f4c1a3bd39b3683e900d3c78b1291d22868fc6" Oct 30 09:29:59.783148 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:29:59.783133 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq" Oct 30 09:30:00.373614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.356310 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e0e5389-b8e8-4033-a7b3-98830bf147af" path="/var/lib/kubelet/pods/6e0e5389-b8e8-4033-a7b3-98830bf147af/volumes" Oct 30 09:30:00.374452 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.374424 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv"] Oct 30 09:30:00.375381 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.375352 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6e0e5389-b8e8-4033-a7b3-98830bf147af" containerName="istio-proxy" Oct 30 09:30:00.375541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.375525 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e0e5389-b8e8-4033-a7b3-98830bf147af" containerName="istio-proxy" Oct 30 09:30:00.380251 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.380220 5938 cpu_manager.go:414] "containerMap: removing container" podUID="653b44ab-fdf8-4f1a-bd3c-946173e53afe" containerName="revoke-keys" Oct 30 09:30:00.380460 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.380445 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653b44ab-fdf8-4f1a-bd3c-946173e53afe" containerName="revoke-keys" Oct 30 09:30:00.380877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.380857 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e0e5389-b8e8-4033-a7b3-98830bf147af" containerName="istio-proxy" Oct 30 09:30:00.381031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.381019 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="653b44ab-fdf8-4f1a-bd3c-946173e53afe" containerName="revoke-keys" Oct 30 09:30:00.562619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.562147 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 09:30:00.610616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.593926 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:00.645663 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.635164 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6"] Oct 30 09:30:00.645663 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.641375 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" Oct 30 09:30:00.676842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.663825 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh"] Oct 30 09:30:00.676842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.666022 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" Oct 30 09:30:00.697530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.677038 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:30:00.697530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.690896 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv"] Oct 30 09:30:00.697530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.690957 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt"] Oct 30 09:30:00.719789 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.715769 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" Oct 30 09:30:00.772927 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.768729 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvpg5\" (UniqueName: \"kubernetes.io/projected/51af45ce-b8eb-44e8-be36-3c66867b161e-kube-api-access-jvpg5\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv\" (UID: \"51af45ce-b8eb-44e8-be36-3c66867b161e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" Oct 30 09:30:00.788519 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.787857 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcrp2\" (UniqueName: \"kubernetes.io/projected/3f8a4ca1-4aa6-4232-af04-d736903ef47f-kube-api-access-xcrp2\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6\" (UID: \"3f8a4ca1-4aa6-4232-af04-d736903ef47f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" Oct 30 09:30:00.788519 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.787988 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8a4ca1-4aa6-4232-af04-d736903ef47f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6\" (UID: \"3f8a4ca1-4aa6-4232-af04-d736903ef47f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" Oct 30 09:30:00.788519 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.788177 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb31bfc-779d-4325-b017-21a3f51a143a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh\" (UID: \"feb31bfc-779d-4325-b017-21a3f51a143a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" Oct 30 09:30:00.788519 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.788248 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwkjh\" (UniqueName: \"kubernetes.io/projected/feb31bfc-779d-4325-b017-21a3f51a143a-kube-api-access-cwkjh\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh\" (UID: \"feb31bfc-779d-4325-b017-21a3f51a143a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" Oct 30 09:30:00.788519 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.788429 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51af45ce-b8eb-44e8-be36-3c66867b161e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv\" (UID: \"51af45ce-b8eb-44e8-be36-3c66867b161e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" Oct 30 09:30:00.925712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.920604 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8a4ca1-4aa6-4232-af04-d736903ef47f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6\" (UID: \"3f8a4ca1-4aa6-4232-af04-d736903ef47f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" Oct 30 09:30:00.925712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.920713 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb31bfc-779d-4325-b017-21a3f51a143a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh\" (UID: \"feb31bfc-779d-4325-b017-21a3f51a143a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" Oct 30 09:30:00.925712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.920755 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwkjh\" (UniqueName: \"kubernetes.io/projected/feb31bfc-779d-4325-b017-21a3f51a143a-kube-api-access-cwkjh\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh\" (UID: \"feb31bfc-779d-4325-b017-21a3f51a143a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" Oct 30 09:30:00.925712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.920863 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51af45ce-b8eb-44e8-be36-3c66867b161e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv\" (UID: \"51af45ce-b8eb-44e8-be36-3c66867b161e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" Oct 30 09:30:00.925712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.921015 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvpg5\" (UniqueName: \"kubernetes.io/projected/51af45ce-b8eb-44e8-be36-3c66867b161e-kube-api-access-jvpg5\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv\" (UID: \"51af45ce-b8eb-44e8-be36-3c66867b161e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" Oct 30 09:30:00.925712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.921070 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcrp2\" (UniqueName: \"kubernetes.io/projected/3f8a4ca1-4aa6-4232-af04-d736903ef47f-kube-api-access-xcrp2\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6\" (UID: \"3f8a4ca1-4aa6-4232-af04-d736903ef47f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" Oct 30 09:30:00.925712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.922386 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8a4ca1-4aa6-4232-af04-d736903ef47f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6\" (UID: \"3f8a4ca1-4aa6-4232-af04-d736903ef47f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" Oct 30 09:30:00.925712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.923204 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb31bfc-779d-4325-b017-21a3f51a143a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh\" (UID: \"feb31bfc-779d-4325-b017-21a3f51a143a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" Oct 30 09:30:00.925712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:00.923524 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51af45ce-b8eb-44e8-be36-3c66867b161e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv\" (UID: \"51af45ce-b8eb-44e8-be36-3c66867b161e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" Oct 30 09:30:01.142615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.136002 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp"] Oct 30 09:30:01.142615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.137810 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" Oct 30 09:30:01.167615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.154157 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6"] Oct 30 09:30:01.167615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.154220 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh"] Oct 30 09:30:01.167615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.154248 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp"] Oct 30 09:30:01.167615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.154272 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt"] Oct 30 09:30:01.167615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.154413 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" Oct 30 09:30:01.199255 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.183684 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:01.207385 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.207342 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwkjh\" (UniqueName: \"kubernetes.io/projected/feb31bfc-779d-4325-b017-21a3f51a143a-kube-api-access-cwkjh\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh\" (UID: \"feb31bfc-779d-4325-b017-21a3f51a143a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" Oct 30 09:30:01.212235 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.212196 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvpg5\" (UniqueName: \"kubernetes.io/projected/51af45ce-b8eb-44e8-be36-3c66867b161e-kube-api-access-jvpg5\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv\" (UID: \"51af45ce-b8eb-44e8-be36-3c66867b161e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" Oct 30 09:30:01.261556 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.259215 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" Oct 30 09:30:01.261556 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.260366 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcrp2\" (UniqueName: \"kubernetes.io/projected/3f8a4ca1-4aa6-4232-af04-d736903ef47f-kube-api-access-xcrp2\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6\" (UID: \"3f8a4ca1-4aa6-4232-af04-d736903ef47f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" Oct 30 09:30:01.292656 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.292509 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c59cd6e-aafa-421b-b299-7ce8f31a7f43-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889210-z8frt\" (UID: \"0c59cd6e-aafa-421b-b299-7ce8f31a7f43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" Oct 30 09:30:01.293423 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.292881 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f407ece5-da79-41f3-9d9e-a3acec3126fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp\" (UID: \"f407ece5-da79-41f3-9d9e-a3acec3126fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" Oct 30 09:30:01.293423 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.292978 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-942v2\" (UniqueName: \"kubernetes.io/projected/0c59cd6e-aafa-421b-b299-7ce8f31a7f43-kube-api-access-942v2\") pod \"maas-api-key-cleanup-29889210-z8frt\" (UID: \"0c59cd6e-aafa-421b-b299-7ce8f31a7f43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" Oct 30 09:30:01.293423 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.293034 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgk4g\" (UniqueName: \"kubernetes.io/projected/f407ece5-da79-41f3-9d9e-a3acec3126fa-kube-api-access-bgk4g\") pod \"maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp\" (UID: \"f407ece5-da79-41f3-9d9e-a3acec3126fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" Oct 30 09:30:01.395866 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.394408 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-942v2\" (UniqueName: \"kubernetes.io/projected/0c59cd6e-aafa-421b-b299-7ce8f31a7f43-kube-api-access-942v2\") pod \"maas-api-key-cleanup-29889210-z8frt\" (UID: \"0c59cd6e-aafa-421b-b299-7ce8f31a7f43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" Oct 30 09:30:01.395866 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.395013 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgk4g\" (UniqueName: \"kubernetes.io/projected/f407ece5-da79-41f3-9d9e-a3acec3126fa-kube-api-access-bgk4g\") pod \"maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp\" (UID: \"f407ece5-da79-41f3-9d9e-a3acec3126fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" Oct 30 09:30:01.395866 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.395339 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c59cd6e-aafa-421b-b299-7ce8f31a7f43-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889210-z8frt\" (UID: \"0c59cd6e-aafa-421b-b299-7ce8f31a7f43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" Oct 30 09:30:01.395866 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.395490 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f407ece5-da79-41f3-9d9e-a3acec3126fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp\" (UID: \"f407ece5-da79-41f3-9d9e-a3acec3126fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" Oct 30 09:30:01.396351 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.396265 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c59cd6e-aafa-421b-b299-7ce8f31a7f43-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889210-z8frt\" (UID: \"0c59cd6e-aafa-421b-b299-7ce8f31a7f43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" Oct 30 09:30:01.396518 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.396493 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f407ece5-da79-41f3-9d9e-a3acec3126fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp\" (UID: \"f407ece5-da79-41f3-9d9e-a3acec3126fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" Oct 30 09:30:01.416333 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.416299 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgk4g\" (UniqueName: \"kubernetes.io/projected/f407ece5-da79-41f3-9d9e-a3acec3126fa-kube-api-access-bgk4g\") pod \"maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp\" (UID: \"f407ece5-da79-41f3-9d9e-a3acec3126fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" Oct 30 09:30:01.416503 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.416357 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-942v2\" (UniqueName: \"kubernetes.io/projected/0c59cd6e-aafa-421b-b299-7ce8f31a7f43-kube-api-access-942v2\") pod \"maas-api-key-cleanup-29889210-z8frt\" (UID: \"0c59cd6e-aafa-421b-b299-7ce8f31a7f43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" Oct 30 09:30:01.464850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.455901 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" Oct 30 09:30:01.484347 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.484127 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" Oct 30 09:30:01.650637 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.646608 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" Oct 30 09:30:01.650637 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.649262 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" Oct 30 09:30:01.652780 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.651694 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh"] Oct 30 09:30:01.820665 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.820332 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv"] Oct 30 09:30:01.821286 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:30:01.821255 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51af45ce_b8eb_44e8_be36_3c66867b161e.slice/crio-bff8939ce9ddcbb34ef0ea95f31b7d49ed7883a1a0ee63fec25b26a9216ca118 WatchSource:0}: Error finding container bff8939ce9ddcbb34ef0ea95f31b7d49ed7883a1a0ee63fec25b26a9216ca118: Status 404 returned error can't find the container with id bff8939ce9ddcbb34ef0ea95f31b7d49ed7883a1a0ee63fec25b26a9216ca118 Oct 30 09:30:01.853162 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.852065 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6"] Oct 30 09:30:01.856120 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:30:01.856081 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8a4ca1_4aa6_4232_af04_d736903ef47f.slice/crio-179e8eb367bdd138f92445e5acd465fdec4b244779976b21038573aec388797b WatchSource:0}: Error finding container 179e8eb367bdd138f92445e5acd465fdec4b244779976b21038573aec388797b: Status 404 returned error can't find the container with id 179e8eb367bdd138f92445e5acd465fdec4b244779976b21038573aec388797b Oct 30 09:30:01.989495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:01.989463 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt"] Oct 30 09:30:02.024023 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:30:02.023322 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c59cd6e_aafa_421b_b299_7ce8f31a7f43.slice/crio-ab7f688dee11b73c62fba6b90a06a2feabe5299bdc635bfbed686ea49a16db30 WatchSource:0}: Error finding container ab7f688dee11b73c62fba6b90a06a2feabe5299bdc635bfbed686ea49a16db30: Status 404 returned error can't find the container with id ab7f688dee11b73c62fba6b90a06a2feabe5299bdc635bfbed686ea49a16db30 Oct 30 09:30:02.028548 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:02.028520 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp"] Oct 30 09:30:02.035202 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:30:02.035169 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf407ece5_da79_41f3_9d9e_a3acec3126fa.slice/crio-3703fe585e26f0426c2e897bc5e8346518581bbd3a1d3dd3383c4f1b1da53b44 WatchSource:0}: Error finding container 3703fe585e26f0426c2e897bc5e8346518581bbd3a1d3dd3383c4f1b1da53b44: Status 404 returned error can't find the container with id 3703fe585e26f0426c2e897bc5e8346518581bbd3a1d3dd3383c4f1b1da53b44 Oct 30 09:30:02.055382 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:02.055347 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" event={"ID":"f407ece5-da79-41f3-9d9e-a3acec3126fa","Type":"ContainerStarted","Data":"3703fe585e26f0426c2e897bc5e8346518581bbd3a1d3dd3383c4f1b1da53b44"} Oct 30 09:30:02.057160 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:02.057132 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" event={"ID":"3f8a4ca1-4aa6-4232-af04-d736903ef47f","Type":"ContainerStarted","Data":"179e8eb367bdd138f92445e5acd465fdec4b244779976b21038573aec388797b"} Oct 30 09:30:02.058922 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:02.058891 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" event={"ID":"51af45ce-b8eb-44e8-be36-3c66867b161e","Type":"ContainerStarted","Data":"bff8939ce9ddcbb34ef0ea95f31b7d49ed7883a1a0ee63fec25b26a9216ca118"} Oct 30 09:30:02.061686 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:02.061637 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" event={"ID":"feb31bfc-779d-4325-b017-21a3f51a143a","Type":"ContainerStarted","Data":"c731d351567a1ce9866bbbabfcef0b7173c440b1ea8a14b3fedc58f44fa72250"} Oct 30 09:30:02.061819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:02.061717 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" event={"ID":"feb31bfc-779d-4325-b017-21a3f51a143a","Type":"ContainerStarted","Data":"7eb98b36e7d1700dfff5b14b183ca84af5426c5d2eda0ec946b8a51b61453a41"} Oct 30 09:30:02.063530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:02.063496 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" event={"ID":"0c59cd6e-aafa-421b-b299-7ce8f31a7f43","Type":"ContainerStarted","Data":"ab7f688dee11b73c62fba6b90a06a2feabe5299bdc635bfbed686ea49a16db30"} Oct 30 09:30:02.103983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:02.103926 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" podStartSLOduration=2.103907694 podStartE2EDuration="2.103907694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:02.101352334 +0000 UTC m=+2119.964058146" watchObservedRunningTime="2026-10-30 09:30:02.103907694 +0000 UTC m=+2119.966613561" Oct 30 09:30:03.073315 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:03.073276 5938 generic.go:356] "Generic (PLEG): container finished" podID="3f8a4ca1-4aa6-4232-af04-d736903ef47f" containerID="3e7db199fab016df40d48d11ac4953d465c1e684c5ecd40e064601559da56dac" exitCode=60 Oct 30 09:30:03.073879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:03.073359 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" event={"ID":"3f8a4ca1-4aa6-4232-af04-d736903ef47f","Type":"ContainerDied","Data":"3e7db199fab016df40d48d11ac4953d465c1e684c5ecd40e064601559da56dac"} Oct 30 09:30:03.074487 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:03.074457 5938 scope.go:122] "RemoveContainer" containerID="3e7db199fab016df40d48d11ac4953d465c1e684c5ecd40e064601559da56dac" Oct 30 09:30:03.075474 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:03.075435 5938 generic.go:356] "Generic (PLEG): container finished" podID="51af45ce-b8eb-44e8-be36-3c66867b161e" containerID="13998a833df126f83fa9efc3a347a48f9af06e83fe1280a26ea6fa0e54bafe2d" exitCode=60 Oct 30 09:30:03.075600 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:03.075475 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" event={"ID":"51af45ce-b8eb-44e8-be36-3c66867b161e","Type":"ContainerDied","Data":"13998a833df126f83fa9efc3a347a48f9af06e83fe1280a26ea6fa0e54bafe2d"} Oct 30 09:30:03.076851 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:03.075878 5938 scope.go:122] "RemoveContainer" containerID="13998a833df126f83fa9efc3a347a48f9af06e83fe1280a26ea6fa0e54bafe2d" Oct 30 09:30:03.079328 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:03.079146 5938 generic.go:356] "Generic (PLEG): container finished" podID="feb31bfc-779d-4325-b017-21a3f51a143a" containerID="c731d351567a1ce9866bbbabfcef0b7173c440b1ea8a14b3fedc58f44fa72250" exitCode=60 Oct 30 09:30:03.079328 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:03.079185 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" event={"ID":"feb31bfc-779d-4325-b017-21a3f51a143a","Type":"ContainerDied","Data":"c731d351567a1ce9866bbbabfcef0b7173c440b1ea8a14b3fedc58f44fa72250"} Oct 30 09:30:03.079571 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:03.079496 5938 scope.go:122] "RemoveContainer" containerID="c731d351567a1ce9866bbbabfcef0b7173c440b1ea8a14b3fedc58f44fa72250" Oct 30 09:30:03.082133 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:03.081569 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" event={"ID":"0c59cd6e-aafa-421b-b299-7ce8f31a7f43","Type":"ContainerStarted","Data":"8421ebe7186aee73d205b0c441998f77fe4223f22f956011324ed077048a5b30"} Oct 30 09:30:03.083554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:03.083523 5938 generic.go:356] "Generic (PLEG): container finished" podID="f407ece5-da79-41f3-9d9e-a3acec3126fa" containerID="bd423046adcfa325599443a4a2eacef31facc044cf719b371118923afd137af3" exitCode=60 Oct 30 09:30:03.083720 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:03.083573 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" event={"ID":"f407ece5-da79-41f3-9d9e-a3acec3126fa","Type":"ContainerDied","Data":"bd423046adcfa325599443a4a2eacef31facc044cf719b371118923afd137af3"} Oct 30 09:30:03.084003 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:03.083983 5938 scope.go:122] "RemoveContainer" containerID="bd423046adcfa325599443a4a2eacef31facc044cf719b371118923afd137af3" Oct 30 09:30:03.153377 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:03.153303 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" podStartSLOduration=3.153285385 podStartE2EDuration="3.153285385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:03.152171912 +0000 UTC m=+2121.014877730" watchObservedRunningTime="2026-10-30 09:30:03.153285385 +0000 UTC m=+2121.015991206" Oct 30 09:30:04.093602 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:04.093484 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" event={"ID":"51af45ce-b8eb-44e8-be36-3c66867b161e","Type":"ContainerStarted","Data":"b356efffe991b091d86574f414778dbee65ee1168b0a7c3ffbaab936000be742"} Oct 30 09:30:04.096065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:04.096026 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" event={"ID":"feb31bfc-779d-4325-b017-21a3f51a143a","Type":"ContainerStarted","Data":"bd423849ae9b1fe72ec4463618b6298326512935dcc4619d7ccc52ffda669f67"} Oct 30 09:30:04.101184 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:04.101150 5938 generic.go:356] "Generic (PLEG): container finished" podID="0c59cd6e-aafa-421b-b299-7ce8f31a7f43" containerID="8421ebe7186aee73d205b0c441998f77fe4223f22f956011324ed077048a5b30" exitCode=60 Oct 30 09:30:04.101369 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:04.101272 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" event={"ID":"0c59cd6e-aafa-421b-b299-7ce8f31a7f43","Type":"ContainerDied","Data":"8421ebe7186aee73d205b0c441998f77fe4223f22f956011324ed077048a5b30"} Oct 30 09:30:04.101554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:04.101534 5938 scope.go:122] "RemoveContainer" containerID="8421ebe7186aee73d205b0c441998f77fe4223f22f956011324ed077048a5b30" Oct 30 09:30:04.103869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:04.103841 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" event={"ID":"f407ece5-da79-41f3-9d9e-a3acec3126fa","Type":"ContainerStarted","Data":"9c946614315aa30aba80f891602be1e095aafd87eef4c79fab8a176dd3176297"} Oct 30 09:30:04.106339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:04.106309 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" event={"ID":"3f8a4ca1-4aa6-4232-af04-d736903ef47f","Type":"ContainerStarted","Data":"bf51b300d9a292861cb4f326d5dd91bc5e7325115c97faf9a20d0be5187bc972"} Oct 30 09:30:04.141746 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:04.141690 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" podStartSLOduration=4.141672824 podStartE2EDuration="4.141672824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:04.139081371 +0000 UTC m=+2122.001787193" watchObservedRunningTime="2026-10-30 09:30:04.141672824 +0000 UTC m=+2122.004378645" Oct 30 09:30:04.182334 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:04.182239 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" podStartSLOduration=4.182223687 podStartE2EDuration="4.182223687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:04.179879658 +0000 UTC m=+2122.042585493" watchObservedRunningTime="2026-10-30 09:30:04.182223687 +0000 UTC m=+2122.044929508" Oct 30 09:30:04.223123 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:04.223062 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" podStartSLOduration=4.223044011 podStartE2EDuration="4.223044011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:04.220250172 +0000 UTC m=+2122.082955984" watchObservedRunningTime="2026-10-30 09:30:04.223044011 +0000 UTC m=+2122.085749832" Oct 30 09:30:05.115201 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.115150 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" event={"ID":"0c59cd6e-aafa-421b-b299-7ce8f31a7f43","Type":"ContainerStarted","Data":"4f9d8886131f7f9053a50797c8a83eb9495d62446c947752c10bdf804444efa5"} Oct 30 09:30:05.117521 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.117487 5938 generic.go:356] "Generic (PLEG): container finished" podID="f407ece5-da79-41f3-9d9e-a3acec3126fa" containerID="9c946614315aa30aba80f891602be1e095aafd87eef4c79fab8a176dd3176297" exitCode=60 Oct 30 09:30:05.117697 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.117572 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" event={"ID":"f407ece5-da79-41f3-9d9e-a3acec3126fa","Type":"ContainerDied","Data":"9c946614315aa30aba80f891602be1e095aafd87eef4c79fab8a176dd3176297"} Oct 30 09:30:05.117801 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.117699 5938 scope.go:122] "RemoveContainer" containerID="bd423046adcfa325599443a4a2eacef31facc044cf719b371118923afd137af3" Oct 30 09:30:05.117923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.117898 5938 scope.go:122] "RemoveContainer" containerID="9c946614315aa30aba80f891602be1e095aafd87eef4c79fab8a176dd3176297" Oct 30 09:30:05.118289 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:05.118262 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp_odh-ai-gateway-infra(f407ece5-da79-41f3-9d9e-a3acec3126fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" podUID="f407ece5-da79-41f3-9d9e-a3acec3126fa" Oct 30 09:30:05.119863 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.119841 5938 generic.go:356] "Generic (PLEG): container finished" podID="3f8a4ca1-4aa6-4232-af04-d736903ef47f" containerID="bf51b300d9a292861cb4f326d5dd91bc5e7325115c97faf9a20d0be5187bc972" exitCode=60 Oct 30 09:30:05.119955 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.119924 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" event={"ID":"3f8a4ca1-4aa6-4232-af04-d736903ef47f","Type":"ContainerDied","Data":"bf51b300d9a292861cb4f326d5dd91bc5e7325115c97faf9a20d0be5187bc972"} Oct 30 09:30:05.120274 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.120260 5938 scope.go:122] "RemoveContainer" containerID="bf51b300d9a292861cb4f326d5dd91bc5e7325115c97faf9a20d0be5187bc972" Oct 30 09:30:05.120571 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:05.120554 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6_odh-ai-gateway-infra(3f8a4ca1-4aa6-4232-af04-d736903ef47f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" podUID="3f8a4ca1-4aa6-4232-af04-d736903ef47f" Oct 30 09:30:05.123018 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.122987 5938 generic.go:356] "Generic (PLEG): container finished" podID="51af45ce-b8eb-44e8-be36-3c66867b161e" containerID="b356efffe991b091d86574f414778dbee65ee1168b0a7c3ffbaab936000be742" exitCode=60 Oct 30 09:30:05.123200 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.123087 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" event={"ID":"51af45ce-b8eb-44e8-be36-3c66867b161e","Type":"ContainerDied","Data":"b356efffe991b091d86574f414778dbee65ee1168b0a7c3ffbaab936000be742"} Oct 30 09:30:05.123367 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.123346 5938 scope.go:122] "RemoveContainer" containerID="b356efffe991b091d86574f414778dbee65ee1168b0a7c3ffbaab936000be742" Oct 30 09:30:05.123753 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:05.123719 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv_odh-ai-gateway-infra(51af45ce-b8eb-44e8-be36-3c66867b161e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" podUID="51af45ce-b8eb-44e8-be36-3c66867b161e" Oct 30 09:30:05.125575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.125552 5938 generic.go:356] "Generic (PLEG): container finished" podID="feb31bfc-779d-4325-b017-21a3f51a143a" containerID="bd423849ae9b1fe72ec4463618b6298326512935dcc4619d7ccc52ffda669f67" exitCode=60 Oct 30 09:30:05.125749 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.125624 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" event={"ID":"feb31bfc-779d-4325-b017-21a3f51a143a","Type":"ContainerDied","Data":"bd423849ae9b1fe72ec4463618b6298326512935dcc4619d7ccc52ffda669f67"} Oct 30 09:30:05.126011 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.125992 5938 scope.go:122] "RemoveContainer" containerID="bd423849ae9b1fe72ec4463618b6298326512935dcc4619d7ccc52ffda669f67" Oct 30 09:30:05.127380 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:05.126330 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh_odh-ai-gateway-infra(feb31bfc-779d-4325-b017-21a3f51a143a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" podUID="feb31bfc-779d-4325-b017-21a3f51a143a" Oct 30 09:30:05.133216 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.133189 5938 scope.go:122] "RemoveContainer" containerID="3e7db199fab016df40d48d11ac4953d465c1e684c5ecd40e064601559da56dac" Oct 30 09:30:05.190997 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.190969 5938 scope.go:122] "RemoveContainer" containerID="13998a833df126f83fa9efc3a347a48f9af06e83fe1280a26ea6fa0e54bafe2d" Oct 30 09:30:05.206433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:05.206329 5938 scope.go:122] "RemoveContainer" containerID="c731d351567a1ce9866bbbabfcef0b7173c440b1ea8a14b3fedc58f44fa72250" Oct 30 09:30:06.135432 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:06.135382 5938 scope.go:122] "RemoveContainer" containerID="b356efffe991b091d86574f414778dbee65ee1168b0a7c3ffbaab936000be742" Oct 30 09:30:06.135997 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:06.135868 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv_odh-ai-gateway-infra(51af45ce-b8eb-44e8-be36-3c66867b161e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" podUID="51af45ce-b8eb-44e8-be36-3c66867b161e" Oct 30 09:30:06.137318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:06.137294 5938 scope.go:122] "RemoveContainer" containerID="bd423849ae9b1fe72ec4463618b6298326512935dcc4619d7ccc52ffda669f67" Oct 30 09:30:06.137717 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:06.137693 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh_odh-ai-gateway-infra(feb31bfc-779d-4325-b017-21a3f51a143a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" podUID="feb31bfc-779d-4325-b017-21a3f51a143a" Oct 30 09:30:06.139253 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:06.139222 5938 generic.go:356] "Generic (PLEG): container finished" podID="0c59cd6e-aafa-421b-b299-7ce8f31a7f43" containerID="4f9d8886131f7f9053a50797c8a83eb9495d62446c947752c10bdf804444efa5" exitCode=60 Oct 30 09:30:06.139368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:06.139305 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" event={"ID":"0c59cd6e-aafa-421b-b299-7ce8f31a7f43","Type":"ContainerDied","Data":"4f9d8886131f7f9053a50797c8a83eb9495d62446c947752c10bdf804444efa5"} Oct 30 09:30:06.139435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:06.139376 5938 scope.go:122] "RemoveContainer" containerID="8421ebe7186aee73d205b0c441998f77fe4223f22f956011324ed077048a5b30" Oct 30 09:30:06.139650 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:06.139634 5938 scope.go:122] "RemoveContainer" containerID="4f9d8886131f7f9053a50797c8a83eb9495d62446c947752c10bdf804444efa5" Oct 30 09:30:06.140025 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:06.139996 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889210-z8frt_odh-ai-gateway-infra(0c59cd6e-aafa-421b-b299-7ce8f31a7f43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" podUID="0c59cd6e-aafa-421b-b299-7ce8f31a7f43" Oct 30 09:30:06.146568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:06.146538 5938 scope.go:122] "RemoveContainer" containerID="9c946614315aa30aba80f891602be1e095aafd87eef4c79fab8a176dd3176297" Oct 30 09:30:06.146952 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:06.146925 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp_odh-ai-gateway-infra(f407ece5-da79-41f3-9d9e-a3acec3126fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" podUID="f407ece5-da79-41f3-9d9e-a3acec3126fa" Oct 30 09:30:06.148793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:06.148770 5938 scope.go:122] "RemoveContainer" containerID="bf51b300d9a292861cb4f326d5dd91bc5e7325115c97faf9a20d0be5187bc972" Oct 30 09:30:06.149099 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:06.149078 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6_odh-ai-gateway-infra(3f8a4ca1-4aa6-4232-af04-d736903ef47f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" podUID="3f8a4ca1-4aa6-4232-af04-d736903ef47f" Oct 30 09:30:07.159128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:07.159094 5938 scope.go:122] "RemoveContainer" containerID="4f9d8886131f7f9053a50797c8a83eb9495d62446c947752c10bdf804444efa5" Oct 30 09:30:07.159621 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:07.159528 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889210-z8frt_odh-ai-gateway-infra(0c59cd6e-aafa-421b-b299-7ce8f31a7f43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" podUID="0c59cd6e-aafa-421b-b299-7ce8f31a7f43" Oct 30 09:30:15.252496 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:15.252314 5938 scope.go:122] "RemoveContainer" containerID="9c946614315aa30aba80f891602be1e095aafd87eef4c79fab8a176dd3176297" Oct 30 09:30:15.819140 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:15.819102 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:30:16.234654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:16.233364 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" event={"ID":"f407ece5-da79-41f3-9d9e-a3acec3126fa","Type":"ContainerStarted","Data":"9e80f9804c0543f1b381550a7b3f2f4db790b26e3d3f7f2a23ebc7f6d175be10"} Oct 30 09:30:16.257613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:16.253620 5938 scope.go:122] "RemoveContainer" containerID="bd423849ae9b1fe72ec4463618b6298326512935dcc4619d7ccc52ffda669f67" Oct 30 09:30:16.257613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:16.254209 5938 scope.go:122] "RemoveContainer" containerID="4f9d8886131f7f9053a50797c8a83eb9495d62446c947752c10bdf804444efa5" Oct 30 09:30:16.257613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:16.254303 5938 scope.go:122] "RemoveContainer" containerID="b356efffe991b091d86574f414778dbee65ee1168b0a7c3ffbaab936000be742" Oct 30 09:30:17.245178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:17.245139 5938 generic.go:356] "Generic (PLEG): container finished" podID="f407ece5-da79-41f3-9d9e-a3acec3126fa" containerID="9e80f9804c0543f1b381550a7b3f2f4db790b26e3d3f7f2a23ebc7f6d175be10" exitCode=60 Oct 30 09:30:17.245371 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:17.245298 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" event={"ID":"f407ece5-da79-41f3-9d9e-a3acec3126fa","Type":"ContainerDied","Data":"9e80f9804c0543f1b381550a7b3f2f4db790b26e3d3f7f2a23ebc7f6d175be10"} Oct 30 09:30:17.245371 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:17.245355 5938 scope.go:122] "RemoveContainer" containerID="9c946614315aa30aba80f891602be1e095aafd87eef4c79fab8a176dd3176297" Oct 30 09:30:17.245938 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:17.245915 5938 scope.go:122] "RemoveContainer" containerID="9e80f9804c0543f1b381550a7b3f2f4db790b26e3d3f7f2a23ebc7f6d175be10" Oct 30 09:30:17.246648 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:17.246349 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp_odh-ai-gateway-infra(f407ece5-da79-41f3-9d9e-a3acec3126fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" podUID="f407ece5-da79-41f3-9d9e-a3acec3126fa" Oct 30 09:30:17.249898 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:17.249818 5938 scope.go:122] "RemoveContainer" containerID="bf51b300d9a292861cb4f326d5dd91bc5e7325115c97faf9a20d0be5187bc972" Oct 30 09:30:17.259152 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:17.255764 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" event={"ID":"51af45ce-b8eb-44e8-be36-3c66867b161e","Type":"ContainerStarted","Data":"b208f26815ec4ad6434f0cf39bc47f0abbf97064c9449255babae519e677de91"} Oct 30 09:30:17.265615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:17.264904 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" event={"ID":"feb31bfc-779d-4325-b017-21a3f51a143a","Type":"ContainerStarted","Data":"bfd3ce28419ad6cfd129589352edc01eba3e0028f9ca9f36531c5538525c55c7"} Oct 30 09:30:17.268051 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:17.268009 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" event={"ID":"0c59cd6e-aafa-421b-b299-7ce8f31a7f43","Type":"ContainerStarted","Data":"50710c1e8dc34bff941161900bcbd6eeb0d9b492c3f916b7ab9984b248d5768b"} Oct 30 09:30:17.288795 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:17.288759 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp"] Oct 30 09:30:18.276417 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.276382 5938 generic.go:356] "Generic (PLEG): container finished" podID="51af45ce-b8eb-44e8-be36-3c66867b161e" containerID="b208f26815ec4ad6434f0cf39bc47f0abbf97064c9449255babae519e677de91" exitCode=60 Oct 30 09:30:18.276964 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.276459 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" event={"ID":"51af45ce-b8eb-44e8-be36-3c66867b161e","Type":"ContainerDied","Data":"b208f26815ec4ad6434f0cf39bc47f0abbf97064c9449255babae519e677de91"} Oct 30 09:30:18.276964 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.276525 5938 scope.go:122] "RemoveContainer" containerID="b356efffe991b091d86574f414778dbee65ee1168b0a7c3ffbaab936000be742" Oct 30 09:30:18.277056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.276977 5938 scope.go:122] "RemoveContainer" containerID="b208f26815ec4ad6434f0cf39bc47f0abbf97064c9449255babae519e677de91" Oct 30 09:30:18.277397 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:18.277367 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv_odh-ai-gateway-infra(51af45ce-b8eb-44e8-be36-3c66867b161e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" podUID="51af45ce-b8eb-44e8-be36-3c66867b161e" Oct 30 09:30:18.278931 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.278910 5938 generic.go:356] "Generic (PLEG): container finished" podID="feb31bfc-779d-4325-b017-21a3f51a143a" containerID="bfd3ce28419ad6cfd129589352edc01eba3e0028f9ca9f36531c5538525c55c7" exitCode=60 Oct 30 09:30:18.279031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.278986 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" event={"ID":"feb31bfc-779d-4325-b017-21a3f51a143a","Type":"ContainerDied","Data":"bfd3ce28419ad6cfd129589352edc01eba3e0028f9ca9f36531c5538525c55c7"} Oct 30 09:30:18.279425 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.279409 5938 scope.go:122] "RemoveContainer" containerID="bfd3ce28419ad6cfd129589352edc01eba3e0028f9ca9f36531c5538525c55c7" Oct 30 09:30:18.279852 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:18.279831 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh_odh-ai-gateway-infra(feb31bfc-779d-4325-b017-21a3f51a143a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" podUID="feb31bfc-779d-4325-b017-21a3f51a143a" Oct 30 09:30:18.281174 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.281151 5938 generic.go:356] "Generic (PLEG): container finished" podID="0c59cd6e-aafa-421b-b299-7ce8f31a7f43" containerID="50710c1e8dc34bff941161900bcbd6eeb0d9b492c3f916b7ab9984b248d5768b" exitCode=60 Oct 30 09:30:18.281276 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.281198 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" event={"ID":"0c59cd6e-aafa-421b-b299-7ce8f31a7f43","Type":"ContainerDied","Data":"50710c1e8dc34bff941161900bcbd6eeb0d9b492c3f916b7ab9984b248d5768b"} Oct 30 09:30:18.281691 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.281673 5938 scope.go:122] "RemoveContainer" containerID="50710c1e8dc34bff941161900bcbd6eeb0d9b492c3f916b7ab9984b248d5768b" Oct 30 09:30:18.282038 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:18.282012 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889210-z8frt_odh-ai-gateway-infra(0c59cd6e-aafa-421b-b299-7ce8f31a7f43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" podUID="0c59cd6e-aafa-421b-b299-7ce8f31a7f43" Oct 30 09:30:18.285992 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.285953 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" event={"ID":"3f8a4ca1-4aa6-4232-af04-d736903ef47f","Type":"ContainerStarted","Data":"4013356bb9b33897eb8c09a886bbf4f22b99eb523c01ba24dc9f441f423589ea"} Oct 30 09:30:18.291093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.291068 5938 scope.go:122] "RemoveContainer" containerID="bd423849ae9b1fe72ec4463618b6298326512935dcc4619d7ccc52ffda669f67" Oct 30 09:30:18.309278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.309257 5938 scope.go:122] "RemoveContainer" containerID="4f9d8886131f7f9053a50797c8a83eb9495d62446c947752c10bdf804444efa5" Oct 30 09:30:18.349399 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.349366 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt"] Oct 30 09:30:18.401906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.401870 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv"] Oct 30 09:30:18.444621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.444566 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh"] Oct 30 09:30:18.493455 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.493425 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" Oct 30 09:30:18.659052 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.659007 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f407ece5-da79-41f3-9d9e-a3acec3126fa-kube-api-access-bgk4g\" (UniqueName: \"kubernetes.io/projected/f407ece5-da79-41f3-9d9e-a3acec3126fa-kube-api-access-bgk4g\") pod \"f407ece5-da79-41f3-9d9e-a3acec3126fa\" (UID: \"f407ece5-da79-41f3-9d9e-a3acec3126fa\") " Oct 30 09:30:18.659250 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.659196 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f407ece5-da79-41f3-9d9e-a3acec3126fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f407ece5-da79-41f3-9d9e-a3acec3126fa-maas-api-service-ca\") pod \"f407ece5-da79-41f3-9d9e-a3acec3126fa\" (UID: \"f407ece5-da79-41f3-9d9e-a3acec3126fa\") " Oct 30 09:30:18.659602 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.659558 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f407ece5-da79-41f3-9d9e-a3acec3126fa-maas-api-service-ca" pod "f407ece5-da79-41f3-9d9e-a3acec3126fa" (UID: "f407ece5-da79-41f3-9d9e-a3acec3126fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:18.659825 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.659807 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f407ece5-da79-41f3-9d9e-a3acec3126fa-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:30:18.661507 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.661483 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f407ece5-da79-41f3-9d9e-a3acec3126fa-kube-api-access-bgk4g" pod "f407ece5-da79-41f3-9d9e-a3acec3126fa" (UID: "f407ece5-da79-41f3-9d9e-a3acec3126fa"). InnerVolumeSpecName "kube-api-access-bgk4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:18.766973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:18.763658 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgk4g\" (UniqueName: \"kubernetes.io/projected/f407ece5-da79-41f3-9d9e-a3acec3126fa-kube-api-access-bgk4g\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:30:19.298871 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.298845 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" Oct 30 09:30:19.299292 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.298873 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp" event={"ID":"f407ece5-da79-41f3-9d9e-a3acec3126fa","Type":"ContainerDied","Data":"3703fe585e26f0426c2e897bc5e8346518581bbd3a1d3dd3383c4f1b1da53b44"} Oct 30 09:30:19.299292 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.298950 5938 scope.go:122] "RemoveContainer" containerID="9e80f9804c0543f1b381550a7b3f2f4db790b26e3d3f7f2a23ebc7f6d175be10" Oct 30 09:30:19.301205 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.301179 5938 generic.go:356] "Generic (PLEG): container finished" podID="3f8a4ca1-4aa6-4232-af04-d736903ef47f" containerID="4013356bb9b33897eb8c09a886bbf4f22b99eb523c01ba24dc9f441f423589ea" exitCode=60 Oct 30 09:30:19.301335 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.301254 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" event={"ID":"3f8a4ca1-4aa6-4232-af04-d736903ef47f","Type":"ContainerDied","Data":"4013356bb9b33897eb8c09a886bbf4f22b99eb523c01ba24dc9f441f423589ea"} Oct 30 09:30:19.301852 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.301831 5938 scope.go:122] "RemoveContainer" containerID="4013356bb9b33897eb8c09a886bbf4f22b99eb523c01ba24dc9f441f423589ea" Oct 30 09:30:19.302231 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:19.302208 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6_odh-ai-gateway-infra(3f8a4ca1-4aa6-4232-af04-d736903ef47f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" podUID="3f8a4ca1-4aa6-4232-af04-d736903ef47f" Oct 30 09:30:19.313027 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.312995 5938 scope.go:122] "RemoveContainer" containerID="bf51b300d9a292861cb4f326d5dd91bc5e7325115c97faf9a20d0be5187bc972" Oct 30 09:30:19.442342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.442282 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp"] Oct 30 09:30:19.446541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.446510 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-c9232f-29889210-pknqp"] Oct 30 09:30:19.461664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.461627 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6"] Oct 30 09:30:19.504068 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.504038 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" Oct 30 09:30:19.575346 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.575317 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" Oct 30 09:30:19.582994 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.582967 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" Oct 30 09:30:19.677197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.677161 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb31bfc-779d-4325-b017-21a3f51a143a-kube-api-access-cwkjh\" (UniqueName: \"kubernetes.io/projected/feb31bfc-779d-4325-b017-21a3f51a143a-kube-api-access-cwkjh\") pod \"feb31bfc-779d-4325-b017-21a3f51a143a\" (UID: \"feb31bfc-779d-4325-b017-21a3f51a143a\") " Oct 30 09:30:19.677378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.677213 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51af45ce-b8eb-44e8-be36-3c66867b161e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51af45ce-b8eb-44e8-be36-3c66867b161e-maas-api-service-ca\") pod \"51af45ce-b8eb-44e8-be36-3c66867b161e\" (UID: \"51af45ce-b8eb-44e8-be36-3c66867b161e\") " Oct 30 09:30:19.677378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.677249 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51af45ce-b8eb-44e8-be36-3c66867b161e-kube-api-access-jvpg5\" (UniqueName: \"kubernetes.io/projected/51af45ce-b8eb-44e8-be36-3c66867b161e-kube-api-access-jvpg5\") pod \"51af45ce-b8eb-44e8-be36-3c66867b161e\" (UID: \"51af45ce-b8eb-44e8-be36-3c66867b161e\") " Oct 30 09:30:19.677515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.677384 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c59cd6e-aafa-421b-b299-7ce8f31a7f43-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c59cd6e-aafa-421b-b299-7ce8f31a7f43-maas-api-service-ca\") pod \"0c59cd6e-aafa-421b-b299-7ce8f31a7f43\" (UID: \"0c59cd6e-aafa-421b-b299-7ce8f31a7f43\") " Oct 30 09:30:19.677515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.677438 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feb31bfc-779d-4325-b017-21a3f51a143a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb31bfc-779d-4325-b017-21a3f51a143a-maas-api-service-ca\") pod \"feb31bfc-779d-4325-b017-21a3f51a143a\" (UID: \"feb31bfc-779d-4325-b017-21a3f51a143a\") " Oct 30 09:30:19.677515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.677474 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c59cd6e-aafa-421b-b299-7ce8f31a7f43-kube-api-access-942v2\" (UniqueName: \"kubernetes.io/projected/0c59cd6e-aafa-421b-b299-7ce8f31a7f43-kube-api-access-942v2\") pod \"0c59cd6e-aafa-421b-b299-7ce8f31a7f43\" (UID: \"0c59cd6e-aafa-421b-b299-7ce8f31a7f43\") " Oct 30 09:30:19.677747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.677657 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51af45ce-b8eb-44e8-be36-3c66867b161e-maas-api-service-ca" pod "51af45ce-b8eb-44e8-be36-3c66867b161e" (UID: "51af45ce-b8eb-44e8-be36-3c66867b161e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:19.677893 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.677861 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c59cd6e-aafa-421b-b299-7ce8f31a7f43-maas-api-service-ca" pod "0c59cd6e-aafa-421b-b299-7ce8f31a7f43" (UID: "0c59cd6e-aafa-421b-b299-7ce8f31a7f43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:19.677893 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.677876 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb31bfc-779d-4325-b017-21a3f51a143a-maas-api-service-ca" pod "feb31bfc-779d-4325-b017-21a3f51a143a" (UID: "feb31bfc-779d-4325-b017-21a3f51a143a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:19.678175 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.678150 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c59cd6e-aafa-421b-b299-7ce8f31a7f43-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:30:19.678280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.678181 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb31bfc-779d-4325-b017-21a3f51a143a-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:30:19.678280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.678195 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51af45ce-b8eb-44e8-be36-3c66867b161e-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:30:19.680662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.680627 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb31bfc-779d-4325-b017-21a3f51a143a-kube-api-access-cwkjh" pod "feb31bfc-779d-4325-b017-21a3f51a143a" (UID: "feb31bfc-779d-4325-b017-21a3f51a143a"). InnerVolumeSpecName "kube-api-access-cwkjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:19.680794 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.680668 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c59cd6e-aafa-421b-b299-7ce8f31a7f43-kube-api-access-942v2" pod "0c59cd6e-aafa-421b-b299-7ce8f31a7f43" (UID: "0c59cd6e-aafa-421b-b299-7ce8f31a7f43"). InnerVolumeSpecName "kube-api-access-942v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:19.680794 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.680690 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51af45ce-b8eb-44e8-be36-3c66867b161e-kube-api-access-jvpg5" pod "51af45ce-b8eb-44e8-be36-3c66867b161e" (UID: "51af45ce-b8eb-44e8-be36-3c66867b161e"). InnerVolumeSpecName "kube-api-access-jvpg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:19.779919 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.779884 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-942v2\" (UniqueName: \"kubernetes.io/projected/0c59cd6e-aafa-421b-b299-7ce8f31a7f43-kube-api-access-942v2\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:30:19.779919 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.779915 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwkjh\" (UniqueName: \"kubernetes.io/projected/feb31bfc-779d-4325-b017-21a3f51a143a-kube-api-access-cwkjh\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:30:19.779919 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:19.779927 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvpg5\" (UniqueName: \"kubernetes.io/projected/51af45ce-b8eb-44e8-be36-3c66867b161e-kube-api-access-jvpg5\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:30:20.257929 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.257898 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f407ece5-da79-41f3-9d9e-a3acec3126fa" path="/var/lib/kubelet/pods/f407ece5-da79-41f3-9d9e-a3acec3126fa/volumes" Oct 30 09:30:20.322195 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.322156 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" Oct 30 09:30:20.322564 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.322156 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv" event={"ID":"51af45ce-b8eb-44e8-be36-3c66867b161e","Type":"ContainerDied","Data":"bff8939ce9ddcbb34ef0ea95f31b7d49ed7883a1a0ee63fec25b26a9216ca118"} Oct 30 09:30:20.322564 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.322316 5938 scope.go:122] "RemoveContainer" containerID="b208f26815ec4ad6434f0cf39bc47f0abbf97064c9449255babae519e677de91" Oct 30 09:30:20.323823 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.323803 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" Oct 30 09:30:20.323968 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.323817 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh" event={"ID":"feb31bfc-779d-4325-b017-21a3f51a143a","Type":"ContainerDied","Data":"7eb98b36e7d1700dfff5b14b183ca84af5426c5d2eda0ec946b8a51b61453a41"} Oct 30 09:30:20.337553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.337503 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" event={"ID":"0c59cd6e-aafa-421b-b299-7ce8f31a7f43","Type":"ContainerDied","Data":"ab7f688dee11b73c62fba6b90a06a2feabe5299bdc635bfbed686ea49a16db30"} Oct 30 09:30:20.337784 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.337557 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt" Oct 30 09:30:20.343403 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.343380 5938 scope.go:122] "RemoveContainer" containerID="bfd3ce28419ad6cfd129589352edc01eba3e0028f9ca9f36531c5538525c55c7" Oct 30 09:30:20.353834 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.353791 5938 scope.go:122] "RemoveContainer" containerID="50710c1e8dc34bff941161900bcbd6eeb0d9b492c3f916b7ab9984b248d5768b" Oct 30 09:30:20.413983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.413952 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt"] Oct 30 09:30:20.425652 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.424811 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889210-z8frt"] Oct 30 09:30:20.461790 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.461540 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv"] Oct 30 09:30:20.464258 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.464227 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889210-gxwcv"] Oct 30 09:30:20.504823 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.504792 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh"] Oct 30 09:30:20.507504 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.507475 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889210-hplxh"] Oct 30 09:30:20.526371 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.526292 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" Oct 30 09:30:20.696921 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.696882 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f8a4ca1-4aa6-4232-af04-d736903ef47f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8a4ca1-4aa6-4232-af04-d736903ef47f-maas-api-service-ca\") pod \"3f8a4ca1-4aa6-4232-af04-d736903ef47f\" (UID: \"3f8a4ca1-4aa6-4232-af04-d736903ef47f\") " Oct 30 09:30:20.697107 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.696962 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f8a4ca1-4aa6-4232-af04-d736903ef47f-kube-api-access-xcrp2\" (UniqueName: \"kubernetes.io/projected/3f8a4ca1-4aa6-4232-af04-d736903ef47f-kube-api-access-xcrp2\") pod \"3f8a4ca1-4aa6-4232-af04-d736903ef47f\" (UID: \"3f8a4ca1-4aa6-4232-af04-d736903ef47f\") " Oct 30 09:30:20.697343 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.697310 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f8a4ca1-4aa6-4232-af04-d736903ef47f-maas-api-service-ca" pod "3f8a4ca1-4aa6-4232-af04-d736903ef47f" (UID: "3f8a4ca1-4aa6-4232-af04-d736903ef47f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:20.699388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.699361 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8a4ca1-4aa6-4232-af04-d736903ef47f-kube-api-access-xcrp2" pod "3f8a4ca1-4aa6-4232-af04-d736903ef47f" (UID: "3f8a4ca1-4aa6-4232-af04-d736903ef47f"). InnerVolumeSpecName "kube-api-access-xcrp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:20.798973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.798880 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8a4ca1-4aa6-4232-af04-d736903ef47f-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:30:20.798973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:20.798913 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcrp2\" (UniqueName: \"kubernetes.io/projected/3f8a4ca1-4aa6-4232-af04-d736903ef47f-kube-api-access-xcrp2\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:30:21.346037 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:21.346006 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" Oct 30 09:30:21.346682 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:21.346003 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6" event={"ID":"3f8a4ca1-4aa6-4232-af04-d736903ef47f","Type":"ContainerDied","Data":"179e8eb367bdd138f92445e5acd465fdec4b244779976b21038573aec388797b"} Oct 30 09:30:21.346682 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:21.346182 5938 scope.go:122] "RemoveContainer" containerID="4013356bb9b33897eb8c09a886bbf4f22b99eb523c01ba24dc9f441f423589ea" Oct 30 09:30:21.403072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:21.403034 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6"] Oct 30 09:30:21.406077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:21.406047 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889210-5vcn6"] Oct 30 09:30:22.257874 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:22.257833 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c59cd6e-aafa-421b-b299-7ce8f31a7f43" path="/var/lib/kubelet/pods/0c59cd6e-aafa-421b-b299-7ce8f31a7f43/volumes" Oct 30 09:30:22.258306 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:22.258292 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f8a4ca1-4aa6-4232-af04-d736903ef47f" path="/var/lib/kubelet/pods/3f8a4ca1-4aa6-4232-af04-d736903ef47f/volumes" Oct 30 09:30:22.258729 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:22.258714 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51af45ce-b8eb-44e8-be36-3c66867b161e" path="/var/lib/kubelet/pods/51af45ce-b8eb-44e8-be36-3c66867b161e/volumes" Oct 30 09:30:22.259452 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:22.259436 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb31bfc-779d-4325-b017-21a3f51a143a" path="/var/lib/kubelet/pods/feb31bfc-779d-4325-b017-21a3f51a143a/volumes" Oct 30 09:30:31.634840 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.634804 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm"] Oct 30 09:30:31.635355 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635340 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f407ece5-da79-41f3-9d9e-a3acec3126fa" containerName="cleanup" Oct 30 09:30:31.635401 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635356 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f407ece5-da79-41f3-9d9e-a3acec3126fa" containerName="cleanup" Oct 30 09:30:31.635401 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635372 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0c59cd6e-aafa-421b-b299-7ce8f31a7f43" containerName="cleanup" Oct 30 09:30:31.635401 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635377 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c59cd6e-aafa-421b-b299-7ce8f31a7f43" containerName="cleanup" Oct 30 09:30:31.635401 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635390 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0c59cd6e-aafa-421b-b299-7ce8f31a7f43" containerName="cleanup" Oct 30 09:30:31.635401 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635395 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c59cd6e-aafa-421b-b299-7ce8f31a7f43" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635405 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f407ece5-da79-41f3-9d9e-a3acec3126fa" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635410 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f407ece5-da79-41f3-9d9e-a3acec3126fa" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635422 5938 cpu_manager.go:414] "containerMap: removing container" podUID="51af45ce-b8eb-44e8-be36-3c66867b161e" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635426 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51af45ce-b8eb-44e8-be36-3c66867b161e" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635432 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f407ece5-da79-41f3-9d9e-a3acec3126fa" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635437 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f407ece5-da79-41f3-9d9e-a3acec3126fa" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635456 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3f8a4ca1-4aa6-4232-af04-d736903ef47f" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635460 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8a4ca1-4aa6-4232-af04-d736903ef47f" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635472 5938 cpu_manager.go:414] "containerMap: removing container" podUID="feb31bfc-779d-4325-b017-21a3f51a143a" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635476 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb31bfc-779d-4325-b017-21a3f51a143a" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635481 5938 cpu_manager.go:414] "containerMap: removing container" podUID="feb31bfc-779d-4325-b017-21a3f51a143a" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635485 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb31bfc-779d-4325-b017-21a3f51a143a" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635503 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3f8a4ca1-4aa6-4232-af04-d736903ef47f" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635508 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8a4ca1-4aa6-4232-af04-d736903ef47f" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635520 5938 cpu_manager.go:414] "containerMap: removing container" podUID="51af45ce-b8eb-44e8-be36-3c66867b161e" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635527 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51af45ce-b8eb-44e8-be36-3c66867b161e" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635535 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3f8a4ca1-4aa6-4232-af04-d736903ef47f" containerName="cleanup" Oct 30 09:30:31.635554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635539 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8a4ca1-4aa6-4232-af04-d736903ef47f" containerName="cleanup" Oct 30 09:30:31.636093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635646 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f407ece5-da79-41f3-9d9e-a3acec3126fa" containerName="cleanup" Oct 30 09:30:31.636093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635656 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c59cd6e-aafa-421b-b299-7ce8f31a7f43" containerName="cleanup" Oct 30 09:30:31.636093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635666 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c59cd6e-aafa-421b-b299-7ce8f31a7f43" containerName="cleanup" Oct 30 09:30:31.636093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635672 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f407ece5-da79-41f3-9d9e-a3acec3126fa" containerName="cleanup" Oct 30 09:30:31.636093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635679 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="51af45ce-b8eb-44e8-be36-3c66867b161e" containerName="cleanup" Oct 30 09:30:31.636093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635684 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f407ece5-da79-41f3-9d9e-a3acec3126fa" containerName="cleanup" Oct 30 09:30:31.636093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635691 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8a4ca1-4aa6-4232-af04-d736903ef47f" containerName="cleanup" Oct 30 09:30:31.636093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635700 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb31bfc-779d-4325-b017-21a3f51a143a" containerName="cleanup" Oct 30 09:30:31.636093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635711 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb31bfc-779d-4325-b017-21a3f51a143a" containerName="cleanup" Oct 30 09:30:31.636093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635718 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb31bfc-779d-4325-b017-21a3f51a143a" containerName="cleanup" Oct 30 09:30:31.636093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635725 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="51af45ce-b8eb-44e8-be36-3c66867b161e" containerName="cleanup" Oct 30 09:30:31.636093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.635731 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8a4ca1-4aa6-4232-af04-d736903ef47f" containerName="cleanup" Oct 30 09:30:31.640809 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.640784 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm" Oct 30 09:30:31.657004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.656972 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm"] Oct 30 09:30:31.667771 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.667727 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:30:31.667933 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.667786 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:30:31.792329 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.792285 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvxmw\" (UniqueName: \"kubernetes.io/projected/0031e67c-37b6-4b1d-ba17-a560d4aafe47-kube-api-access-cvxmw\") pod \"maas-api-revoke-sub-561cd2e4-kxptm\" (UID: \"0031e67c-37b6-4b1d-ba17-a560d4aafe47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm" Oct 30 09:30:31.792329 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.792334 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0031e67c-37b6-4b1d-ba17-a560d4aafe47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-561cd2e4-kxptm\" (UID: \"0031e67c-37b6-4b1d-ba17-a560d4aafe47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm" Oct 30 09:30:31.792571 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.792402 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0031e67c-37b6-4b1d-ba17-a560d4aafe47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-561cd2e4-kxptm\" (UID: \"0031e67c-37b6-4b1d-ba17-a560d4aafe47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm" Oct 30 09:30:31.894095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.894003 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxmw\" (UniqueName: \"kubernetes.io/projected/0031e67c-37b6-4b1d-ba17-a560d4aafe47-kube-api-access-cvxmw\") pod \"maas-api-revoke-sub-561cd2e4-kxptm\" (UID: \"0031e67c-37b6-4b1d-ba17-a560d4aafe47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm" Oct 30 09:30:31.894095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.894042 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0031e67c-37b6-4b1d-ba17-a560d4aafe47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-561cd2e4-kxptm\" (UID: \"0031e67c-37b6-4b1d-ba17-a560d4aafe47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm" Oct 30 09:30:31.894095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.894081 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0031e67c-37b6-4b1d-ba17-a560d4aafe47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-561cd2e4-kxptm\" (UID: \"0031e67c-37b6-4b1d-ba17-a560d4aafe47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm" Oct 30 09:30:31.894930 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.894866 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0031e67c-37b6-4b1d-ba17-a560d4aafe47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-561cd2e4-kxptm\" (UID: \"0031e67c-37b6-4b1d-ba17-a560d4aafe47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm" Oct 30 09:30:31.896859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.896836 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0031e67c-37b6-4b1d-ba17-a560d4aafe47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-561cd2e4-kxptm\" (UID: \"0031e67c-37b6-4b1d-ba17-a560d4aafe47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm" Oct 30 09:30:31.917088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.917045 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxmw\" (UniqueName: \"kubernetes.io/projected/0031e67c-37b6-4b1d-ba17-a560d4aafe47-kube-api-access-cvxmw\") pod \"maas-api-revoke-sub-561cd2e4-kxptm\" (UID: \"0031e67c-37b6-4b1d-ba17-a560d4aafe47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm" Oct 30 09:30:31.956124 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:31.956086 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm" Oct 30 09:30:32.264000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:32.263973 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm"] Oct 30 09:30:32.588018 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:32.587982 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm" event={"ID":"0031e67c-37b6-4b1d-ba17-a560d4aafe47","Type":"ContainerStarted","Data":"1527d726306102ff483ed01ca2bf3c472d1d72eb7005d788fa3448e30a4305fb"} Oct 30 09:30:33.597630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:33.597520 5938 generic.go:356] "Generic (PLEG): container finished" podID="0031e67c-37b6-4b1d-ba17-a560d4aafe47" containerID="22852ba1efc96d9bd4c280eeca1ba1f16f83d54c242eb5c6f8b2be3cf2dcf130" exitCode=0 Oct 30 09:30:33.597630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:33.597579 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm" event={"ID":"0031e67c-37b6-4b1d-ba17-a560d4aafe47","Type":"ContainerDied","Data":"22852ba1efc96d9bd4c280eeca1ba1f16f83d54c242eb5c6f8b2be3cf2dcf130"} Oct 30 09:30:34.763433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:34.763400 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm" Oct 30 09:30:34.835217 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:34.835176 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0031e67c-37b6-4b1d-ba17-a560d4aafe47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0031e67c-37b6-4b1d-ba17-a560d4aafe47-maas-api-xks-ca\") pod \"0031e67c-37b6-4b1d-ba17-a560d4aafe47\" (UID: \"0031e67c-37b6-4b1d-ba17-a560d4aafe47\") " Oct 30 09:30:34.835447 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:34.835255 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0031e67c-37b6-4b1d-ba17-a560d4aafe47-kube-api-access-cvxmw\" (UniqueName: \"kubernetes.io/projected/0031e67c-37b6-4b1d-ba17-a560d4aafe47-kube-api-access-cvxmw\") pod \"0031e67c-37b6-4b1d-ba17-a560d4aafe47\" (UID: \"0031e67c-37b6-4b1d-ba17-a560d4aafe47\") " Oct 30 09:30:34.835447 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:34.835361 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0031e67c-37b6-4b1d-ba17-a560d4aafe47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0031e67c-37b6-4b1d-ba17-a560d4aafe47-maas-api-openshift-ca\") pod \"0031e67c-37b6-4b1d-ba17-a560d4aafe47\" (UID: \"0031e67c-37b6-4b1d-ba17-a560d4aafe47\") " Oct 30 09:30:34.835807 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:34.835767 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0031e67c-37b6-4b1d-ba17-a560d4aafe47-maas-api-openshift-ca" pod "0031e67c-37b6-4b1d-ba17-a560d4aafe47" (UID: "0031e67c-37b6-4b1d-ba17-a560d4aafe47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:34.836139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:34.836111 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0031e67c-37b6-4b1d-ba17-a560d4aafe47-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:30:34.838322 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:34.838276 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0031e67c-37b6-4b1d-ba17-a560d4aafe47-maas-api-xks-ca" pod "0031e67c-37b6-4b1d-ba17-a560d4aafe47" (UID: "0031e67c-37b6-4b1d-ba17-a560d4aafe47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:30:34.838684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:34.838657 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0031e67c-37b6-4b1d-ba17-a560d4aafe47-kube-api-access-cvxmw" pod "0031e67c-37b6-4b1d-ba17-a560d4aafe47" (UID: "0031e67c-37b6-4b1d-ba17-a560d4aafe47"). InnerVolumeSpecName "kube-api-access-cvxmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:34.937366 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:34.937332 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0031e67c-37b6-4b1d-ba17-a560d4aafe47-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:30:34.937366 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:34.937362 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvxmw\" (UniqueName: \"kubernetes.io/projected/0031e67c-37b6-4b1d-ba17-a560d4aafe47-kube-api-access-cvxmw\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:30:35.614436 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:35.614398 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm" Oct 30 09:30:35.614436 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:35.614418 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm" event={"ID":"0031e67c-37b6-4b1d-ba17-a560d4aafe47","Type":"ContainerDied","Data":"1527d726306102ff483ed01ca2bf3c472d1d72eb7005d788fa3448e30a4305fb"} Oct 30 09:30:35.614689 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:35.614473 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1527d726306102ff483ed01ca2bf3c472d1d72eb7005d788fa3448e30a4305fb" Oct 30 09:30:41.706796 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.706750 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc"] Oct 30 09:30:41.707677 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.707651 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0031e67c-37b6-4b1d-ba17-a560d4aafe47" containerName="revoke-keys" Oct 30 09:30:41.707677 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.707677 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0031e67c-37b6-4b1d-ba17-a560d4aafe47" containerName="revoke-keys" Oct 30 09:30:41.707842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.707698 5938 cpu_manager.go:414] "containerMap: removing container" podUID="feb31bfc-779d-4325-b017-21a3f51a143a" containerName="cleanup" Oct 30 09:30:41.707842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.707706 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb31bfc-779d-4325-b017-21a3f51a143a" containerName="cleanup" Oct 30 09:30:41.707842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.707743 5938 cpu_manager.go:414] "containerMap: removing container" podUID="51af45ce-b8eb-44e8-be36-3c66867b161e" containerName="cleanup" Oct 30 09:30:41.707842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.707750 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51af45ce-b8eb-44e8-be36-3c66867b161e" containerName="cleanup" Oct 30 09:30:41.707842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.707778 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0c59cd6e-aafa-421b-b299-7ce8f31a7f43" containerName="cleanup" Oct 30 09:30:41.707842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.707785 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c59cd6e-aafa-421b-b299-7ce8f31a7f43" containerName="cleanup" Oct 30 09:30:41.708071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.707930 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="51af45ce-b8eb-44e8-be36-3c66867b161e" containerName="cleanup" Oct 30 09:30:41.708071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.707950 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0031e67c-37b6-4b1d-ba17-a560d4aafe47" containerName="revoke-keys" Oct 30 09:30:41.708071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.707966 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8a4ca1-4aa6-4232-af04-d736903ef47f" containerName="cleanup" Oct 30 09:30:41.708071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.707979 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c59cd6e-aafa-421b-b299-7ce8f31a7f43" containerName="cleanup" Oct 30 09:30:41.712502 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.712472 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.736552 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.736507 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc"] Oct 30 09:30:41.747886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.747851 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.747886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.747893 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.748161 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.748042 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.748161 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.748069 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.748161 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.748119 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.748161 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.748152 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.748351 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.748181 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqcm8\" (UniqueName: \"kubernetes.io/projected/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-kube-api-access-xqcm8\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.748351 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.748260 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.748351 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.748326 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.849737 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.849701 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.849934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.849779 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.849934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.849800 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.849934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.849873 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.849934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.849895 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.850143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.849936 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.850143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.849968 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.850143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.849999 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqcm8\" (UniqueName: \"kubernetes.io/projected/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-kube-api-access-xqcm8\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.850143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.850093 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.850341 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.850259 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.850576 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.850542 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.850738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.850579 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.850811 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.850724 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.850811 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.850796 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.852552 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.852527 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.852695 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.852681 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.870669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.870626 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:41.870827 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:41.870756 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqcm8\" (UniqueName: \"kubernetes.io/projected/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-kube-api-access-xqcm8\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:42.040119 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:42.035783 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:42.383167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:42.381537 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc"] Oct 30 09:30:42.385619 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:30:42.385566 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa20f8e_6be0_484a_a363_28bbe0aa6d88.slice/crio-bcda93dd789c893838130f3b9a2b170b516b4e6d5d45a1ee1ac91069ee4ce2bd WatchSource:0}: Error finding container bcda93dd789c893838130f3b9a2b170b516b4e6d5d45a1ee1ac91069ee4ce2bd: Status 404 returned error can't find the container with id bcda93dd789c893838130f3b9a2b170b516b4e6d5d45a1ee1ac91069ee4ce2bd Oct 30 09:30:42.388812 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:42.388758 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:30:42.388962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:42.388852 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:30:42.388962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:42.388884 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:30:42.670088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:42.669996 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" event={"ID":"3aa20f8e-6be0-484a-a363-28bbe0aa6d88","Type":"ContainerStarted","Data":"d34cff2ccf703aab7192adf5699bc305b19824f44074030efbb0b5f08779c024"} Oct 30 09:30:42.670088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:42.670050 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" event={"ID":"3aa20f8e-6be0-484a-a363-28bbe0aa6d88","Type":"ContainerStarted","Data":"bcda93dd789c893838130f3b9a2b170b516b4e6d5d45a1ee1ac91069ee4ce2bd"} Oct 30 09:30:42.721856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:42.721792 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" podStartSLOduration=1.7217739380000001 podStartE2EDuration="1.721773938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:42.719260635 +0000 UTC m=+2160.581966470" watchObservedRunningTime="2026-10-30 09:30:42.721773938 +0000 UTC m=+2160.584479866" Oct 30 09:30:42.911457 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:42.911417 5938 scope.go:122] "RemoveContainer" containerID="7e437cafd311955f2b446862532a3472307d24d1bc2d5f23b4b04ce09c0debc0" Oct 30 09:30:43.040612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:43.037772 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:43.043402 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:43.043363 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 09:30:43.043688 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:43.043431 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" podUID="3aa20f8e-6be0-484a-a363-28bbe0aa6d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 09:30:44.039614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:44.039073 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 09:30:44.039614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:44.039153 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" podUID="3aa20f8e-6be0-484a-a363-28bbe0aa6d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 09:30:45.043736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:45.042744 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 09:30:45.043736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:45.042819 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" podUID="3aa20f8e-6be0-484a-a363-28bbe0aa6d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 09:30:45.811187 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:30:45.811145 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:30:45.968966 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:45.968922 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh"] Oct 30 09:30:45.975121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:45.975087 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" Oct 30 09:30:45.994707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:45.994673 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh"] Oct 30 09:30:46.010849 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.010807 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07951f59-63ef-491b-9bb7-24038d2c5c5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb699118-vjbrh\" (UID: \"07951f59-63ef-491b-9bb7-24038d2c5c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" Oct 30 09:30:46.011088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.010959 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07951f59-63ef-491b-9bb7-24038d2c5c5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb699118-vjbrh\" (UID: \"07951f59-63ef-491b-9bb7-24038d2c5c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" Oct 30 09:30:46.011088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.011000 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6jtc\" (UniqueName: \"kubernetes.io/projected/07951f59-63ef-491b-9bb7-24038d2c5c5c-kube-api-access-j6jtc\") pod \"maas-api-revoke-sub-cb699118-vjbrh\" (UID: \"07951f59-63ef-491b-9bb7-24038d2c5c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" Oct 30 09:30:46.050734 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.050696 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 09:30:46.051240 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.050770 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" podUID="3aa20f8e-6be0-484a-a363-28bbe0aa6d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 09:30:46.084159 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.084063 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:30:46.084420 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.084077 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:30:46.112664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.112631 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07951f59-63ef-491b-9bb7-24038d2c5c5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb699118-vjbrh\" (UID: \"07951f59-63ef-491b-9bb7-24038d2c5c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" Oct 30 09:30:46.112996 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.112771 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07951f59-63ef-491b-9bb7-24038d2c5c5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb699118-vjbrh\" (UID: \"07951f59-63ef-491b-9bb7-24038d2c5c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" Oct 30 09:30:46.112996 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.112812 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6jtc\" (UniqueName: \"kubernetes.io/projected/07951f59-63ef-491b-9bb7-24038d2c5c5c-kube-api-access-j6jtc\") pod \"maas-api-revoke-sub-cb699118-vjbrh\" (UID: \"07951f59-63ef-491b-9bb7-24038d2c5c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" Oct 30 09:30:46.113459 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.113433 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07951f59-63ef-491b-9bb7-24038d2c5c5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb699118-vjbrh\" (UID: \"07951f59-63ef-491b-9bb7-24038d2c5c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" Oct 30 09:30:46.117336 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.117304 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07951f59-63ef-491b-9bb7-24038d2c5c5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb699118-vjbrh\" (UID: \"07951f59-63ef-491b-9bb7-24038d2c5c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" Oct 30 09:30:46.152832 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.152793 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6jtc\" (UniqueName: \"kubernetes.io/projected/07951f59-63ef-491b-9bb7-24038d2c5c5c-kube-api-access-j6jtc\") pod \"maas-api-revoke-sub-cb699118-vjbrh\" (UID: \"07951f59-63ef-491b-9bb7-24038d2c5c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" Oct 30 09:30:46.288428 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.288386 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql"] Oct 30 09:30:46.292927 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.292901 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" Oct 30 09:30:46.296634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.296597 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" Oct 30 09:30:46.310992 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.310921 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-066f95\"" Oct 30 09:30:46.311115 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.311001 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-066f95-dockercfg-bsg87\"" Oct 30 09:30:46.336969 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.319111 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql"] Oct 30 09:30:46.433654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.433561 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f465c90-efbc-4ce7-9dba-bf62492777c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql\" (UID: \"0f465c90-efbc-4ce7-9dba-bf62492777c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" Oct 30 09:30:46.433876 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.433723 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggms6\" (UniqueName: \"kubernetes.io/projected/0f465c90-efbc-4ce7-9dba-bf62492777c9-kube-api-access-ggms6\") pod \"payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql\" (UID: \"0f465c90-efbc-4ce7-9dba-bf62492777c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" Oct 30 09:30:46.508617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.496658 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t"] Oct 30 09:30:46.515112 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.515084 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t"] Oct 30 09:30:46.515460 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.515442 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" Oct 30 09:30:46.543881 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.536621 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f465c90-efbc-4ce7-9dba-bf62492777c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql\" (UID: \"0f465c90-efbc-4ce7-9dba-bf62492777c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" Oct 30 09:30:46.543881 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.536701 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggms6\" (UniqueName: \"kubernetes.io/projected/0f465c90-efbc-4ce7-9dba-bf62492777c9-kube-api-access-ggms6\") pod \"payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql\" (UID: \"0f465c90-efbc-4ce7-9dba-bf62492777c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" Oct 30 09:30:46.543881 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.540340 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f465c90-efbc-4ce7-9dba-bf62492777c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql\" (UID: \"0f465c90-efbc-4ce7-9dba-bf62492777c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" Oct 30 09:30:46.577254 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.577219 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggms6\" (UniqueName: \"kubernetes.io/projected/0f465c90-efbc-4ce7-9dba-bf62492777c9-kube-api-access-ggms6\") pod \"payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql\" (UID: \"0f465c90-efbc-4ce7-9dba-bf62492777c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" Oct 30 09:30:46.615054 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.615018 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" Oct 30 09:30:46.640684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.639508 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc771d3-8e16-4e67-ab32-8c25d4326e86-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t\" (UID: \"ecc771d3-8e16-4e67-ab32-8c25d4326e86\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" Oct 30 09:30:46.640684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.639682 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f562v\" (UniqueName: \"kubernetes.io/projected/ecc771d3-8e16-4e67-ab32-8c25d4326e86-kube-api-access-f562v\") pod \"payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t\" (UID: \"ecc771d3-8e16-4e67-ab32-8c25d4326e86\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" Oct 30 09:30:46.749432 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.743431 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc771d3-8e16-4e67-ab32-8c25d4326e86-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t\" (UID: \"ecc771d3-8e16-4e67-ab32-8c25d4326e86\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" Oct 30 09:30:46.749432 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.747655 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f562v\" (UniqueName: \"kubernetes.io/projected/ecc771d3-8e16-4e67-ab32-8c25d4326e86-kube-api-access-f562v\") pod \"payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t\" (UID: \"ecc771d3-8e16-4e67-ab32-8c25d4326e86\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" Oct 30 09:30:46.749432 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.749370 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc771d3-8e16-4e67-ab32-8c25d4326e86-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t\" (UID: \"ecc771d3-8e16-4e67-ab32-8c25d4326e86\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" Oct 30 09:30:46.769465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.769429 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f562v\" (UniqueName: \"kubernetes.io/projected/ecc771d3-8e16-4e67-ab32-8c25d4326e86-kube-api-access-f562v\") pod \"payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t\" (UID: \"ecc771d3-8e16-4e67-ab32-8c25d4326e86\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" Oct 30 09:30:46.847009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.846954 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh"] Oct 30 09:30:46.849909 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:30:46.849868 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07951f59_63ef_491b_9bb7_24038d2c5c5c.slice/crio-01cecdf4f732dba9d70273437e31d370fa4dcf95c3ca844fa00c329dd54862f5 WatchSource:0}: Error finding container 01cecdf4f732dba9d70273437e31d370fa4dcf95c3ca844fa00c329dd54862f5: Status 404 returned error can't find the container with id 01cecdf4f732dba9d70273437e31d370fa4dcf95c3ca844fa00c329dd54862f5 Oct 30 09:30:46.873084 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.873058 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" Oct 30 09:30:46.960707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:46.960616 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql"] Oct 30 09:30:47.048200 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:47.047541 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 09:30:47.048200 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:47.047620 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" podUID="3aa20f8e-6be0-484a-a363-28bbe0aa6d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 09:30:47.201834 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:47.201805 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t"] Oct 30 09:30:47.204731 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:30:47.204693 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc771d3_8e16_4e67_ab32_8c25d4326e86.slice/crio-386257aa93d640d281926c3c14d8b555095faf805f0f32a344abfa16a0511dcd WatchSource:0}: Error finding container 386257aa93d640d281926c3c14d8b555095faf805f0f32a344abfa16a0511dcd: Status 404 returned error can't find the container with id 386257aa93d640d281926c3c14d8b555095faf805f0f32a344abfa16a0511dcd Oct 30 09:30:47.716018 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:47.715971 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" event={"ID":"0f465c90-efbc-4ce7-9dba-bf62492777c9","Type":"ContainerStarted","Data":"798296cfe1871e2575395771675a30e27926211b984de6ce40e9eff4490dd8e5"} Oct 30 09:30:47.716254 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:47.716086 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" event={"ID":"0f465c90-efbc-4ce7-9dba-bf62492777c9","Type":"ContainerStarted","Data":"1758fd440467c644be4c77b97e433b778206e249e32282a2dd9e31519d316be7"} Oct 30 09:30:47.716254 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:47.716188 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" Oct 30 09:30:47.718293 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:47.718254 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" event={"ID":"ecc771d3-8e16-4e67-ab32-8c25d4326e86","Type":"ContainerStarted","Data":"a740f460989e6bac23e448d3e52153c081564229825b7ac9652e36aa7701f11c"} Oct 30 09:30:47.718469 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:47.718322 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" Oct 30 09:30:47.718469 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:47.718349 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" event={"ID":"ecc771d3-8e16-4e67-ab32-8c25d4326e86","Type":"ContainerStarted","Data":"386257aa93d640d281926c3c14d8b555095faf805f0f32a344abfa16a0511dcd"} Oct 30 09:30:47.745360 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:47.745308 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" event={"ID":"07951f59-63ef-491b-9bb7-24038d2c5c5c","Type":"ContainerStarted","Data":"06f7df41f0a1a397337be80f612ee5f9e97759757b8c7a19e3cc39d71f2bc5c8"} Oct 30 09:30:47.745547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:47.745384 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" event={"ID":"07951f59-63ef-491b-9bb7-24038d2c5c5c","Type":"ContainerStarted","Data":"01cecdf4f732dba9d70273437e31d370fa4dcf95c3ca844fa00c329dd54862f5"} Oct 30 09:30:47.764360 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:47.764301 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" podStartSLOduration=1.764285107 podStartE2EDuration="1.764285107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:47.760260317 +0000 UTC m=+2165.622966138" watchObservedRunningTime="2026-10-30 09:30:47.764285107 +0000 UTC m=+2165.626990958" Oct 30 09:30:47.801196 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:47.801133 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" podStartSLOduration=2.801116424 podStartE2EDuration="2.801116424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:47.800544773 +0000 UTC m=+2165.663250596" watchObservedRunningTime="2026-10-30 09:30:47.801116424 +0000 UTC m=+2165.663822245" Oct 30 09:30:47.844628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:47.844268 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" podStartSLOduration=1.8442453749999999 podStartE2EDuration="1.844245375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:47.842574462 +0000 UTC m=+2165.705280285" watchObservedRunningTime="2026-10-30 09:30:47.844245375 +0000 UTC m=+2165.706951198" Oct 30 09:30:48.048782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:48.048726 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 09:30:48.048941 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:48.048796 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" podUID="3aa20f8e-6be0-484a-a363-28bbe0aa6d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 09:30:48.755330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:48.755292 5938 generic.go:356] "Generic (PLEG): container finished" podID="07951f59-63ef-491b-9bb7-24038d2c5c5c" containerID="06f7df41f0a1a397337be80f612ee5f9e97759757b8c7a19e3cc39d71f2bc5c8" exitCode=0 Oct 30 09:30:48.755775 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:48.755356 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" event={"ID":"07951f59-63ef-491b-9bb7-24038d2c5c5c","Type":"ContainerDied","Data":"06f7df41f0a1a397337be80f612ee5f9e97759757b8c7a19e3cc39d71f2bc5c8"} Oct 30 09:30:49.037284 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:49.037191 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 09:30:49.037284 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:49.037247 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" podUID="3aa20f8e-6be0-484a-a363-28bbe0aa6d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 09:30:49.984251 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:49.984221 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" Oct 30 09:30:49.996181 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:49.996149 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07951f59-63ef-491b-9bb7-24038d2c5c5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07951f59-63ef-491b-9bb7-24038d2c5c5c-maas-api-xks-ca\") pod \"07951f59-63ef-491b-9bb7-24038d2c5c5c\" (UID: \"07951f59-63ef-491b-9bb7-24038d2c5c5c\") " Oct 30 09:30:49.996333 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:49.996194 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07951f59-63ef-491b-9bb7-24038d2c5c5c-kube-api-access-j6jtc\" (UniqueName: \"kubernetes.io/projected/07951f59-63ef-491b-9bb7-24038d2c5c5c-kube-api-access-j6jtc\") pod \"07951f59-63ef-491b-9bb7-24038d2c5c5c\" (UID: \"07951f59-63ef-491b-9bb7-24038d2c5c5c\") " Oct 30 09:30:49.996407 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:49.996395 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07951f59-63ef-491b-9bb7-24038d2c5c5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07951f59-63ef-491b-9bb7-24038d2c5c5c-maas-api-openshift-ca\") pod \"07951f59-63ef-491b-9bb7-24038d2c5c5c\" (UID: \"07951f59-63ef-491b-9bb7-24038d2c5c5c\") " Oct 30 09:30:49.996773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:49.996742 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07951f59-63ef-491b-9bb7-24038d2c5c5c-maas-api-openshift-ca" pod "07951f59-63ef-491b-9bb7-24038d2c5c5c" (UID: "07951f59-63ef-491b-9bb7-24038d2c5c5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:49.998501 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:49.998474 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07951f59-63ef-491b-9bb7-24038d2c5c5c-maas-api-xks-ca" pod "07951f59-63ef-491b-9bb7-24038d2c5c5c" (UID: "07951f59-63ef-491b-9bb7-24038d2c5c5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:30:49.998662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:49.998532 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07951f59-63ef-491b-9bb7-24038d2c5c5c-kube-api-access-j6jtc" pod "07951f59-63ef-491b-9bb7-24038d2c5c5c" (UID: "07951f59-63ef-491b-9bb7-24038d2c5c5c"). InnerVolumeSpecName "kube-api-access-j6jtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:50.042210 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:50.042110 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:50.042578 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:50.042548 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:50.043454 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:50.043429 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:30:50.098708 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:50.098673 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07951f59-63ef-491b-9bb7-24038d2c5c5c-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:30:50.098708 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:50.098702 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6jtc\" (UniqueName: \"kubernetes.io/projected/07951f59-63ef-491b-9bb7-24038d2c5c5c-kube-api-access-j6jtc\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:30:50.098708 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:50.098717 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07951f59-63ef-491b-9bb7-24038d2c5c5c-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:30:50.797088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:50.797058 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" Oct 30 09:30:50.797280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:50.797085 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh" event={"ID":"07951f59-63ef-491b-9bb7-24038d2c5c5c","Type":"ContainerDied","Data":"01cecdf4f732dba9d70273437e31d370fa4dcf95c3ca844fa00c329dd54862f5"} Oct 30 09:30:50.797280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:50.797136 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01cecdf4f732dba9d70273437e31d370fa4dcf95c3ca844fa00c329dd54862f5" Oct 30 09:30:54.241670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:54.240702 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:54.444613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:54.443743 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:54.538681 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:54.537811 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 09:30:54.592613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:54.591903 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:58.759732 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:58.759693 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" Oct 30 09:30:58.760437 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:30:58.760414 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" Oct 30 09:31:00.185486 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.185444 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t"] Oct 30 09:31:00.186032 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.186017 5938 cpu_manager.go:414] "containerMap: removing container" podUID="07951f59-63ef-491b-9bb7-24038d2c5c5c" containerName="revoke-keys" Oct 30 09:31:00.186032 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.186033 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07951f59-63ef-491b-9bb7-24038d2c5c5c" containerName="revoke-keys" Oct 30 09:31:00.186159 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.186148 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="07951f59-63ef-491b-9bb7-24038d2c5c5c" containerName="revoke-keys" Oct 30 09:31:00.190031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.190009 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t" Oct 30 09:31:00.213268 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.213231 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t"] Oct 30 09:31:00.222039 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.222007 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:31:00.222039 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.222035 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:31:00.315512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.315470 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcpm4\" (UniqueName: \"kubernetes.io/projected/d2ab290b-5408-493b-ba9c-0c3a651ec49b-kube-api-access-qcpm4\") pod \"maas-api-revoke-sub-7bf3c8b6-btv9t\" (UID: \"d2ab290b-5408-493b-ba9c-0c3a651ec49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t" Oct 30 09:31:00.315773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.315536 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ab290b-5408-493b-ba9c-0c3a651ec49b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bf3c8b6-btv9t\" (UID: \"d2ab290b-5408-493b-ba9c-0c3a651ec49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t" Oct 30 09:31:00.315773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.315662 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ab290b-5408-493b-ba9c-0c3a651ec49b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bf3c8b6-btv9t\" (UID: \"d2ab290b-5408-493b-ba9c-0c3a651ec49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t" Oct 30 09:31:00.417061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.417017 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcpm4\" (UniqueName: \"kubernetes.io/projected/d2ab290b-5408-493b-ba9c-0c3a651ec49b-kube-api-access-qcpm4\") pod \"maas-api-revoke-sub-7bf3c8b6-btv9t\" (UID: \"d2ab290b-5408-493b-ba9c-0c3a651ec49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t" Oct 30 09:31:00.417263 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.417085 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ab290b-5408-493b-ba9c-0c3a651ec49b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bf3c8b6-btv9t\" (UID: \"d2ab290b-5408-493b-ba9c-0c3a651ec49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t" Oct 30 09:31:00.417263 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.417158 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ab290b-5408-493b-ba9c-0c3a651ec49b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bf3c8b6-btv9t\" (UID: \"d2ab290b-5408-493b-ba9c-0c3a651ec49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t" Oct 30 09:31:00.418028 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.417996 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ab290b-5408-493b-ba9c-0c3a651ec49b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bf3c8b6-btv9t\" (UID: \"d2ab290b-5408-493b-ba9c-0c3a651ec49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t" Oct 30 09:31:00.420402 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.420378 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ab290b-5408-493b-ba9c-0c3a651ec49b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bf3c8b6-btv9t\" (UID: \"d2ab290b-5408-493b-ba9c-0c3a651ec49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t" Oct 30 09:31:00.438418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.438335 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcpm4\" (UniqueName: \"kubernetes.io/projected/d2ab290b-5408-493b-ba9c-0c3a651ec49b-kube-api-access-qcpm4\") pod \"maas-api-revoke-sub-7bf3c8b6-btv9t\" (UID: \"d2ab290b-5408-493b-ba9c-0c3a651ec49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t" Oct 30 09:31:00.509027 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.508995 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t" Oct 30 09:31:00.834133 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.834098 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t"] Oct 30 09:31:00.835475 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:31:00.835445 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ab290b_5408_493b_ba9c_0c3a651ec49b.slice/crio-66f8bfa79d7f86f9630d569cdff80c228d06495b3febee17fca61a100174859e WatchSource:0}: Error finding container 66f8bfa79d7f86f9630d569cdff80c228d06495b3febee17fca61a100174859e: Status 404 returned error can't find the container with id 66f8bfa79d7f86f9630d569cdff80c228d06495b3febee17fca61a100174859e Oct 30 09:31:00.941747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:00.941710 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t" event={"ID":"d2ab290b-5408-493b-ba9c-0c3a651ec49b","Type":"ContainerStarted","Data":"66f8bfa79d7f86f9630d569cdff80c228d06495b3febee17fca61a100174859e"} Oct 30 09:31:01.951747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:01.951714 5938 generic.go:356] "Generic (PLEG): container finished" podID="d2ab290b-5408-493b-ba9c-0c3a651ec49b" containerID="c371df455f76fe5c124c875a439404860e49dd8f711db66e9dbdb7aa5b726492" exitCode=0 Oct 30 09:31:01.952181 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:01.951815 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t" event={"ID":"d2ab290b-5408-493b-ba9c-0c3a651ec49b","Type":"ContainerDied","Data":"c371df455f76fe5c124c875a439404860e49dd8f711db66e9dbdb7aa5b726492"} Oct 30 09:31:03.124144 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:03.124112 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t" Oct 30 09:31:03.260277 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:03.260186 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2ab290b-5408-493b-ba9c-0c3a651ec49b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ab290b-5408-493b-ba9c-0c3a651ec49b-maas-api-openshift-ca\") pod \"d2ab290b-5408-493b-ba9c-0c3a651ec49b\" (UID: \"d2ab290b-5408-493b-ba9c-0c3a651ec49b\") " Oct 30 09:31:03.260463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:03.260318 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2ab290b-5408-493b-ba9c-0c3a651ec49b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ab290b-5408-493b-ba9c-0c3a651ec49b-maas-api-xks-ca\") pod \"d2ab290b-5408-493b-ba9c-0c3a651ec49b\" (UID: \"d2ab290b-5408-493b-ba9c-0c3a651ec49b\") " Oct 30 09:31:03.260463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:03.260427 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ab290b-5408-493b-ba9c-0c3a651ec49b-kube-api-access-qcpm4\" (UniqueName: \"kubernetes.io/projected/d2ab290b-5408-493b-ba9c-0c3a651ec49b-kube-api-access-qcpm4\") pod \"d2ab290b-5408-493b-ba9c-0c3a651ec49b\" (UID: \"d2ab290b-5408-493b-ba9c-0c3a651ec49b\") " Oct 30 09:31:03.260580 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:03.260555 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ab290b-5408-493b-ba9c-0c3a651ec49b-maas-api-openshift-ca" pod "d2ab290b-5408-493b-ba9c-0c3a651ec49b" (UID: "d2ab290b-5408-493b-ba9c-0c3a651ec49b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:03.260948 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:03.260921 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ab290b-5408-493b-ba9c-0c3a651ec49b-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:03.262756 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:03.262729 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ab290b-5408-493b-ba9c-0c3a651ec49b-maas-api-xks-ca" pod "d2ab290b-5408-493b-ba9c-0c3a651ec49b" (UID: "d2ab290b-5408-493b-ba9c-0c3a651ec49b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:31:03.262868 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:03.262765 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ab290b-5408-493b-ba9c-0c3a651ec49b-kube-api-access-qcpm4" pod "d2ab290b-5408-493b-ba9c-0c3a651ec49b" (UID: "d2ab290b-5408-493b-ba9c-0c3a651ec49b"). InnerVolumeSpecName "kube-api-access-qcpm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:03.362453 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:03.362415 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ab290b-5408-493b-ba9c-0c3a651ec49b-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:03.362453 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:03.362451 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcpm4\" (UniqueName: \"kubernetes.io/projected/d2ab290b-5408-493b-ba9c-0c3a651ec49b-kube-api-access-qcpm4\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:03.968159 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:03.968126 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t" Oct 30 09:31:03.968159 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:03.968147 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t" event={"ID":"d2ab290b-5408-493b-ba9c-0c3a651ec49b","Type":"ContainerDied","Data":"66f8bfa79d7f86f9630d569cdff80c228d06495b3febee17fca61a100174859e"} Oct 30 09:31:03.968423 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:03.968237 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66f8bfa79d7f86f9630d569cdff80c228d06495b3febee17fca61a100174859e" Oct 30 09:31:11.683046 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.683011 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp"] Oct 30 09:31:11.683567 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.683553 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d2ab290b-5408-493b-ba9c-0c3a651ec49b" containerName="revoke-keys" Oct 30 09:31:11.683669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.683568 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ab290b-5408-493b-ba9c-0c3a651ec49b" containerName="revoke-keys" Oct 30 09:31:11.683723 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.683712 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ab290b-5408-493b-ba9c-0c3a651ec49b" containerName="revoke-keys" Oct 30 09:31:11.687631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.687582 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" Oct 30 09:31:11.705376 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.705343 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp"] Oct 30 09:31:11.715717 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.715685 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:31:11.715840 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.715745 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:31:11.779363 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.779324 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b218ff4-xb2lp\" (UID: \"09bafb89-2b8a-42bc-a0a4-8ba4eed62526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" Oct 30 09:31:11.779537 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.779484 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b218ff4-xb2lp\" (UID: \"09bafb89-2b8a-42bc-a0a4-8ba4eed62526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" Oct 30 09:31:11.779684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.779667 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6lp5\" (UniqueName: \"kubernetes.io/projected/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-kube-api-access-r6lp5\") pod \"maas-api-revoke-sub-0b218ff4-xb2lp\" (UID: \"09bafb89-2b8a-42bc-a0a4-8ba4eed62526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" Oct 30 09:31:11.881276 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.881238 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6lp5\" (UniqueName: \"kubernetes.io/projected/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-kube-api-access-r6lp5\") pod \"maas-api-revoke-sub-0b218ff4-xb2lp\" (UID: \"09bafb89-2b8a-42bc-a0a4-8ba4eed62526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" Oct 30 09:31:11.881464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.881381 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b218ff4-xb2lp\" (UID: \"09bafb89-2b8a-42bc-a0a4-8ba4eed62526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" Oct 30 09:31:11.881509 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.881473 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b218ff4-xb2lp\" (UID: \"09bafb89-2b8a-42bc-a0a4-8ba4eed62526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" Oct 30 09:31:11.882051 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.882031 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b218ff4-xb2lp\" (UID: \"09bafb89-2b8a-42bc-a0a4-8ba4eed62526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" Oct 30 09:31:11.884095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.884073 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b218ff4-xb2lp\" (UID: \"09bafb89-2b8a-42bc-a0a4-8ba4eed62526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" Oct 30 09:31:11.903264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:11.903230 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6lp5\" (UniqueName: \"kubernetes.io/projected/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-kube-api-access-r6lp5\") pod \"maas-api-revoke-sub-0b218ff4-xb2lp\" (UID: \"09bafb89-2b8a-42bc-a0a4-8ba4eed62526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" Oct 30 09:31:12.001302 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:12.001213 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" Oct 30 09:31:12.307743 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:12.307717 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp"] Oct 30 09:31:12.310763 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:31:12.310733 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bafb89_2b8a_42bc_a0a4_8ba4eed62526.slice/crio-f66bcd86e6b42516f2c03c93881d8c686051fa231419b6ee02ec1dc4d6a8d0af WatchSource:0}: Error finding container f66bcd86e6b42516f2c03c93881d8c686051fa231419b6ee02ec1dc4d6a8d0af: Status 404 returned error can't find the container with id f66bcd86e6b42516f2c03c93881d8c686051fa231419b6ee02ec1dc4d6a8d0af Oct 30 09:31:13.029599 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:13.029481 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" event={"ID":"09bafb89-2b8a-42bc-a0a4-8ba4eed62526","Type":"ContainerStarted","Data":"6853c926b9601b584eff1424dbe1997c1e471578b681399c50853e39efe22249"} Oct 30 09:31:13.029989 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:13.029610 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" event={"ID":"09bafb89-2b8a-42bc-a0a4-8ba4eed62526","Type":"ContainerStarted","Data":"f66bcd86e6b42516f2c03c93881d8c686051fa231419b6ee02ec1dc4d6a8d0af"} Oct 30 09:31:13.070715 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:13.070661 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" podStartSLOduration=2.070643838 podStartE2EDuration="2.070643838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:31:13.069457051 +0000 UTC m=+2190.932162863" watchObservedRunningTime="2026-10-30 09:31:13.070643838 +0000 UTC m=+2190.933349659" Oct 30 09:31:14.037783 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:14.037749 5938 generic.go:356] "Generic (PLEG): container finished" podID="09bafb89-2b8a-42bc-a0a4-8ba4eed62526" containerID="6853c926b9601b584eff1424dbe1997c1e471578b681399c50853e39efe22249" exitCode=0 Oct 30 09:31:14.038192 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:14.037831 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" event={"ID":"09bafb89-2b8a-42bc-a0a4-8ba4eed62526","Type":"ContainerDied","Data":"6853c926b9601b584eff1424dbe1997c1e471578b681399c50853e39efe22249"} Oct 30 09:31:15.283643 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:15.271142 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" Oct 30 09:31:15.333399 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:15.333309 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-maas-api-xks-ca\") pod \"09bafb89-2b8a-42bc-a0a4-8ba4eed62526\" (UID: \"09bafb89-2b8a-42bc-a0a4-8ba4eed62526\") " Oct 30 09:31:15.333626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:15.333428 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-maas-api-openshift-ca\") pod \"09bafb89-2b8a-42bc-a0a4-8ba4eed62526\" (UID: \"09bafb89-2b8a-42bc-a0a4-8ba4eed62526\") " Oct 30 09:31:15.333626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:15.333484 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-kube-api-access-r6lp5\" (UniqueName: \"kubernetes.io/projected/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-kube-api-access-r6lp5\") pod \"09bafb89-2b8a-42bc-a0a4-8ba4eed62526\" (UID: \"09bafb89-2b8a-42bc-a0a4-8ba4eed62526\") " Oct 30 09:31:15.333913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:15.333885 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-maas-api-openshift-ca" pod "09bafb89-2b8a-42bc-a0a4-8ba4eed62526" (UID: "09bafb89-2b8a-42bc-a0a4-8ba4eed62526"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:15.334263 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:15.334242 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:15.336223 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:15.336195 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-maas-api-xks-ca" pod "09bafb89-2b8a-42bc-a0a4-8ba4eed62526" (UID: "09bafb89-2b8a-42bc-a0a4-8ba4eed62526"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:31:15.336887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:15.336866 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-kube-api-access-r6lp5" pod "09bafb89-2b8a-42bc-a0a4-8ba4eed62526" (UID: "09bafb89-2b8a-42bc-a0a4-8ba4eed62526"). InnerVolumeSpecName "kube-api-access-r6lp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:15.436055 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:15.436020 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:15.436055 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:15.436050 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6lp5\" (UniqueName: \"kubernetes.io/projected/09bafb89-2b8a-42bc-a0a4-8ba4eed62526-kube-api-access-r6lp5\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:15.809149 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:31:15.809110 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:31:16.056024 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:16.055971 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" event={"ID":"09bafb89-2b8a-42bc-a0a4-8ba4eed62526","Type":"ContainerDied","Data":"f66bcd86e6b42516f2c03c93881d8c686051fa231419b6ee02ec1dc4d6a8d0af"} Oct 30 09:31:16.056203 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:16.056082 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f66bcd86e6b42516f2c03c93881d8c686051fa231419b6ee02ec1dc4d6a8d0af" Oct 30 09:31:16.056203 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:16.056165 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp" Oct 30 09:31:18.109761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:18.109707 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" podUID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:18.127750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:18.127693 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:19.498143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.498106 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7"] Oct 30 09:31:19.498712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.498696 5938 cpu_manager.go:414] "containerMap: removing container" podUID="09bafb89-2b8a-42bc-a0a4-8ba4eed62526" containerName="revoke-keys" Oct 30 09:31:19.498773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.498713 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09bafb89-2b8a-42bc-a0a4-8ba4eed62526" containerName="revoke-keys" Oct 30 09:31:19.498848 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.498839 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="09bafb89-2b8a-42bc-a0a4-8ba4eed62526" containerName="revoke-keys" Oct 30 09:31:19.502550 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.502530 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7" Oct 30 09:31:19.521026 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.520996 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:31:19.521792 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.521769 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7"] Oct 30 09:31:19.531666 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.531642 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:31:19.549607 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.549560 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a5c890-2gvq7\" (UID: \"f41f71a6-94a3-4f27-8ce0-cba6940bfba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7" Oct 30 09:31:19.549847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.549720 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a5c890-2gvq7\" (UID: \"f41f71a6-94a3-4f27-8ce0-cba6940bfba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7" Oct 30 09:31:19.549847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.549746 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqs67\" (UniqueName: \"kubernetes.io/projected/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-kube-api-access-hqs67\") pod \"maas-api-revoke-sub-54a5c890-2gvq7\" (UID: \"f41f71a6-94a3-4f27-8ce0-cba6940bfba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7" Oct 30 09:31:19.650725 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.650686 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a5c890-2gvq7\" (UID: \"f41f71a6-94a3-4f27-8ce0-cba6940bfba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7" Oct 30 09:31:19.650725 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.650724 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqs67\" (UniqueName: \"kubernetes.io/projected/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-kube-api-access-hqs67\") pod \"maas-api-revoke-sub-54a5c890-2gvq7\" (UID: \"f41f71a6-94a3-4f27-8ce0-cba6940bfba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7" Oct 30 09:31:19.650971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.650884 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a5c890-2gvq7\" (UID: \"f41f71a6-94a3-4f27-8ce0-cba6940bfba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7" Oct 30 09:31:19.651500 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.651481 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a5c890-2gvq7\" (UID: \"f41f71a6-94a3-4f27-8ce0-cba6940bfba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7" Oct 30 09:31:19.653350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.653327 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a5c890-2gvq7\" (UID: \"f41f71a6-94a3-4f27-8ce0-cba6940bfba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7" Oct 30 09:31:19.671927 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.671898 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqs67\" (UniqueName: \"kubernetes.io/projected/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-kube-api-access-hqs67\") pod \"maas-api-revoke-sub-54a5c890-2gvq7\" (UID: \"f41f71a6-94a3-4f27-8ce0-cba6940bfba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7" Oct 30 09:31:19.816622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:19.816515 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7" Oct 30 09:31:20.121563 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:20.121528 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7"] Oct 30 09:31:20.123014 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:31:20.122983 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41f71a6_94a3_4f27_8ce0_cba6940bfba5.slice/crio-e69970ea065bfceffe9cbd0689d2d0a432841d2fdf98db84979a245fb4a2d026 WatchSource:0}: Error finding container e69970ea065bfceffe9cbd0689d2d0a432841d2fdf98db84979a245fb4a2d026: Status 404 returned error can't find the container with id e69970ea065bfceffe9cbd0689d2d0a432841d2fdf98db84979a245fb4a2d026 Oct 30 09:31:20.180723 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:20.180685 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7" event={"ID":"f41f71a6-94a3-4f27-8ce0-cba6940bfba5","Type":"ContainerStarted","Data":"e69970ea065bfceffe9cbd0689d2d0a432841d2fdf98db84979a245fb4a2d026"} Oct 30 09:31:21.197360 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:21.197319 5938 generic.go:356] "Generic (PLEG): container finished" podID="f41f71a6-94a3-4f27-8ce0-cba6940bfba5" containerID="7b88773692a770eea3eb3daf0d51a931722774d79bebdbf6a814190f83680200" exitCode=0 Oct 30 09:31:21.197809 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:21.197399 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7" event={"ID":"f41f71a6-94a3-4f27-8ce0-cba6940bfba5","Type":"ContainerDied","Data":"7b88773692a770eea3eb3daf0d51a931722774d79bebdbf6a814190f83680200"} Oct 30 09:31:22.313178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.313127 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx"] Oct 30 09:31:22.317536 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.317513 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx" Oct 30 09:31:22.342397 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.342360 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx"] Oct 30 09:31:22.376174 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.376151 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7" Oct 30 09:31:22.386665 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.386632 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37e62be0-a6d4-411b-a7ac-b91b066f24ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx\" (UID: \"37e62be0-a6d4-411b-a7ac-b91b066f24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx" Oct 30 09:31:22.386866 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.386834 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqmg7\" (UniqueName: \"kubernetes.io/projected/37e62be0-a6d4-411b-a7ac-b91b066f24ec-kube-api-access-gqmg7\") pod \"maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx\" (UID: \"37e62be0-a6d4-411b-a7ac-b91b066f24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx" Oct 30 09:31:22.387359 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.387335 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37e62be0-a6d4-411b-a7ac-b91b066f24ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx\" (UID: \"37e62be0-a6d4-411b-a7ac-b91b066f24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx" Oct 30 09:31:22.488442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.488353 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-kube-api-access-hqs67\" (UniqueName: \"kubernetes.io/projected/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-kube-api-access-hqs67\") pod \"f41f71a6-94a3-4f27-8ce0-cba6940bfba5\" (UID: \"f41f71a6-94a3-4f27-8ce0-cba6940bfba5\") " Oct 30 09:31:22.488442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.488427 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-maas-api-openshift-ca\") pod \"f41f71a6-94a3-4f27-8ce0-cba6940bfba5\" (UID: \"f41f71a6-94a3-4f27-8ce0-cba6940bfba5\") " Oct 30 09:31:22.488738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.488581 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-maas-api-xks-ca\") pod \"f41f71a6-94a3-4f27-8ce0-cba6940bfba5\" (UID: \"f41f71a6-94a3-4f27-8ce0-cba6940bfba5\") " Oct 30 09:31:22.488878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.488854 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-maas-api-openshift-ca" pod "f41f71a6-94a3-4f27-8ce0-cba6940bfba5" (UID: "f41f71a6-94a3-4f27-8ce0-cba6940bfba5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:22.488953 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.488935 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37e62be0-a6d4-411b-a7ac-b91b066f24ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx\" (UID: \"37e62be0-a6d4-411b-a7ac-b91b066f24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx" Oct 30 09:31:22.489100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.489083 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmg7\" (UniqueName: \"kubernetes.io/projected/37e62be0-a6d4-411b-a7ac-b91b066f24ec-kube-api-access-gqmg7\") pod \"maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx\" (UID: \"37e62be0-a6d4-411b-a7ac-b91b066f24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx" Oct 30 09:31:22.489292 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.489266 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37e62be0-a6d4-411b-a7ac-b91b066f24ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx\" (UID: \"37e62be0-a6d4-411b-a7ac-b91b066f24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx" Oct 30 09:31:22.489422 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.489402 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:22.489750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.489726 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37e62be0-a6d4-411b-a7ac-b91b066f24ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx\" (UID: \"37e62be0-a6d4-411b-a7ac-b91b066f24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx" Oct 30 09:31:22.490945 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.490920 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-kube-api-access-hqs67" pod "f41f71a6-94a3-4f27-8ce0-cba6940bfba5" (UID: "f41f71a6-94a3-4f27-8ce0-cba6940bfba5"). InnerVolumeSpecName "kube-api-access-hqs67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:22.491018 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.490970 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-maas-api-xks-ca" pod "f41f71a6-94a3-4f27-8ce0-cba6940bfba5" (UID: "f41f71a6-94a3-4f27-8ce0-cba6940bfba5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:31:22.491734 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.491714 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37e62be0-a6d4-411b-a7ac-b91b066f24ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx\" (UID: \"37e62be0-a6d4-411b-a7ac-b91b066f24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx" Oct 30 09:31:22.509845 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.509811 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmg7\" (UniqueName: \"kubernetes.io/projected/37e62be0-a6d4-411b-a7ac-b91b066f24ec-kube-api-access-gqmg7\") pod \"maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx\" (UID: \"37e62be0-a6d4-411b-a7ac-b91b066f24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx" Oct 30 09:31:22.590266 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.590229 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:22.590266 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.590262 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqs67\" (UniqueName: \"kubernetes.io/projected/f41f71a6-94a3-4f27-8ce0-cba6940bfba5-kube-api-access-hqs67\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:22.641973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.641932 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx" Oct 30 09:31:22.946654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:22.946615 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx"] Oct 30 09:31:22.947992 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:31:22.947961 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e62be0_a6d4_411b_a7ac_b91b066f24ec.slice/crio-ec764a2883cefe84ef4f494bb15f55aabbefcec7ff8c665cfcaa56aee3d29d86 WatchSource:0}: Error finding container ec764a2883cefe84ef4f494bb15f55aabbefcec7ff8c665cfcaa56aee3d29d86: Status 404 returned error can't find the container with id ec764a2883cefe84ef4f494bb15f55aabbefcec7ff8c665cfcaa56aee3d29d86 Oct 30 09:31:23.211786 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:23.211703 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7" Oct 30 09:31:23.211786 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:23.211717 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7" event={"ID":"f41f71a6-94a3-4f27-8ce0-cba6940bfba5","Type":"ContainerDied","Data":"e69970ea065bfceffe9cbd0689d2d0a432841d2fdf98db84979a245fb4a2d026"} Oct 30 09:31:23.211786 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:23.211782 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e69970ea065bfceffe9cbd0689d2d0a432841d2fdf98db84979a245fb4a2d026" Oct 30 09:31:23.213510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:23.213480 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx" event={"ID":"37e62be0-a6d4-411b-a7ac-b91b066f24ec","Type":"ContainerStarted","Data":"ec764a2883cefe84ef4f494bb15f55aabbefcec7ff8c665cfcaa56aee3d29d86"} Oct 30 09:31:24.222954 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:24.222870 5938 generic.go:356] "Generic (PLEG): container finished" podID="37e62be0-a6d4-411b-a7ac-b91b066f24ec" containerID="34b5d033f094e28c45e93f8e4be855b236bfad41baa7b390dd82692f75a755da" exitCode=0 Oct 30 09:31:24.223361 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:24.222953 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx" event={"ID":"37e62be0-a6d4-411b-a7ac-b91b066f24ec","Type":"ContainerDied","Data":"34b5d033f094e28c45e93f8e4be855b236bfad41baa7b390dd82692f75a755da"} Oct 30 09:31:25.388461 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:25.388421 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx" Oct 30 09:31:25.536894 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:25.536790 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37e62be0-a6d4-411b-a7ac-b91b066f24ec-kube-api-access-gqmg7\" (UniqueName: \"kubernetes.io/projected/37e62be0-a6d4-411b-a7ac-b91b066f24ec-kube-api-access-gqmg7\") pod \"37e62be0-a6d4-411b-a7ac-b91b066f24ec\" (UID: \"37e62be0-a6d4-411b-a7ac-b91b066f24ec\") " Oct 30 09:31:25.537090 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:25.536901 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37e62be0-a6d4-411b-a7ac-b91b066f24ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37e62be0-a6d4-411b-a7ac-b91b066f24ec-maas-api-xks-ca\") pod \"37e62be0-a6d4-411b-a7ac-b91b066f24ec\" (UID: \"37e62be0-a6d4-411b-a7ac-b91b066f24ec\") " Oct 30 09:31:25.537215 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:25.537188 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37e62be0-a6d4-411b-a7ac-b91b066f24ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37e62be0-a6d4-411b-a7ac-b91b066f24ec-maas-api-openshift-ca\") pod \"37e62be0-a6d4-411b-a7ac-b91b066f24ec\" (UID: \"37e62be0-a6d4-411b-a7ac-b91b066f24ec\") " Oct 30 09:31:25.537694 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:25.537514 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e62be0-a6d4-411b-a7ac-b91b066f24ec-maas-api-openshift-ca" pod "37e62be0-a6d4-411b-a7ac-b91b066f24ec" (UID: "37e62be0-a6d4-411b-a7ac-b91b066f24ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:25.537822 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:25.537719 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37e62be0-a6d4-411b-a7ac-b91b066f24ec-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:25.539286 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:25.539258 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e62be0-a6d4-411b-a7ac-b91b066f24ec-maas-api-xks-ca" pod "37e62be0-a6d4-411b-a7ac-b91b066f24ec" (UID: "37e62be0-a6d4-411b-a7ac-b91b066f24ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:31:25.539388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:25.539337 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e62be0-a6d4-411b-a7ac-b91b066f24ec-kube-api-access-gqmg7" pod "37e62be0-a6d4-411b-a7ac-b91b066f24ec" (UID: "37e62be0-a6d4-411b-a7ac-b91b066f24ec"). InnerVolumeSpecName "kube-api-access-gqmg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:25.639406 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:25.639365 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqmg7\" (UniqueName: \"kubernetes.io/projected/37e62be0-a6d4-411b-a7ac-b91b066f24ec-kube-api-access-gqmg7\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:25.639406 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:25.639410 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37e62be0-a6d4-411b-a7ac-b91b066f24ec-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:26.238309 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:26.238266 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx" event={"ID":"37e62be0-a6d4-411b-a7ac-b91b066f24ec","Type":"ContainerDied","Data":"ec764a2883cefe84ef4f494bb15f55aabbefcec7ff8c665cfcaa56aee3d29d86"} Oct 30 09:31:26.238309 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:26.238298 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx" Oct 30 09:31:26.238601 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:26.238319 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec764a2883cefe84ef4f494bb15f55aabbefcec7ff8c665cfcaa56aee3d29d86" Oct 30 09:31:30.161327 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.161286 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm"] Oct 30 09:31:30.161940 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.161922 5938 cpu_manager.go:414] "containerMap: removing container" podUID="37e62be0-a6d4-411b-a7ac-b91b066f24ec" containerName="revoke-keys" Oct 30 09:31:30.161987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.161942 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37e62be0-a6d4-411b-a7ac-b91b066f24ec" containerName="revoke-keys" Oct 30 09:31:30.161987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.161970 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f41f71a6-94a3-4f27-8ce0-cba6940bfba5" containerName="revoke-keys" Oct 30 09:31:30.161987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.161975 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41f71a6-94a3-4f27-8ce0-cba6940bfba5" containerName="revoke-keys" Oct 30 09:31:30.162093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.162072 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="37e62be0-a6d4-411b-a7ac-b91b066f24ec" containerName="revoke-keys" Oct 30 09:31:30.162093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.162085 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41f71a6-94a3-4f27-8ce0-cba6940bfba5" containerName="revoke-keys" Oct 30 09:31:30.165634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.165579 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" Oct 30 09:31:30.207669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.205988 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:31:30.207669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.207190 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:31:30.227144 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.226992 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm"] Oct 30 09:31:30.326761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.326716 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79dc7829-7dcb-4118-a759-ed05d381e749-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57f2560e-24mrm\" (UID: \"79dc7829-7dcb-4118-a759-ed05d381e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" Oct 30 09:31:30.326968 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.326882 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rdwg\" (UniqueName: \"kubernetes.io/projected/79dc7829-7dcb-4118-a759-ed05d381e749-kube-api-access-9rdwg\") pod \"maas-api-revoke-sub-57f2560e-24mrm\" (UID: \"79dc7829-7dcb-4118-a759-ed05d381e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" Oct 30 09:31:30.327022 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.326973 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79dc7829-7dcb-4118-a759-ed05d381e749-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57f2560e-24mrm\" (UID: \"79dc7829-7dcb-4118-a759-ed05d381e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" Oct 30 09:31:30.428228 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.428133 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79dc7829-7dcb-4118-a759-ed05d381e749-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57f2560e-24mrm\" (UID: \"79dc7829-7dcb-4118-a759-ed05d381e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" Oct 30 09:31:30.428438 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.428254 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rdwg\" (UniqueName: \"kubernetes.io/projected/79dc7829-7dcb-4118-a759-ed05d381e749-kube-api-access-9rdwg\") pod \"maas-api-revoke-sub-57f2560e-24mrm\" (UID: \"79dc7829-7dcb-4118-a759-ed05d381e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" Oct 30 09:31:30.428438 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.428400 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79dc7829-7dcb-4118-a759-ed05d381e749-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57f2560e-24mrm\" (UID: \"79dc7829-7dcb-4118-a759-ed05d381e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" Oct 30 09:31:30.429051 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.429021 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79dc7829-7dcb-4118-a759-ed05d381e749-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57f2560e-24mrm\" (UID: \"79dc7829-7dcb-4118-a759-ed05d381e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" Oct 30 09:31:30.431347 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.431325 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79dc7829-7dcb-4118-a759-ed05d381e749-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57f2560e-24mrm\" (UID: \"79dc7829-7dcb-4118-a759-ed05d381e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" Oct 30 09:31:30.452523 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.452495 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rdwg\" (UniqueName: \"kubernetes.io/projected/79dc7829-7dcb-4118-a759-ed05d381e749-kube-api-access-9rdwg\") pod \"maas-api-revoke-sub-57f2560e-24mrm\" (UID: \"79dc7829-7dcb-4118-a759-ed05d381e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" Oct 30 09:31:30.479736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.479697 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" Oct 30 09:31:30.797872 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:30.797841 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm"] Oct 30 09:31:30.798845 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:31:30.798817 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79dc7829_7dcb_4118_a759_ed05d381e749.slice/crio-8f255dff05ee03bfd5be7bc5bc54aa5b39a365114f06474e38251bf6d9081552 WatchSource:0}: Error finding container 8f255dff05ee03bfd5be7bc5bc54aa5b39a365114f06474e38251bf6d9081552: Status 404 returned error can't find the container with id 8f255dff05ee03bfd5be7bc5bc54aa5b39a365114f06474e38251bf6d9081552 Oct 30 09:31:31.276673 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:31.276641 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" event={"ID":"79dc7829-7dcb-4118-a759-ed05d381e749","Type":"ContainerStarted","Data":"71a900597b1f0d399daf7c1768542207ec21a3d8a0dc10cce627dbfbdbe4afb4"} Oct 30 09:31:31.277099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:31.276704 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" event={"ID":"79dc7829-7dcb-4118-a759-ed05d381e749","Type":"ContainerStarted","Data":"8f255dff05ee03bfd5be7bc5bc54aa5b39a365114f06474e38251bf6d9081552"} Oct 30 09:31:31.323809 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:31.323744 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" podStartSLOduration=1.323730342 podStartE2EDuration="1.323730342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:31:31.320907607 +0000 UTC m=+2209.183613428" watchObservedRunningTime="2026-10-30 09:31:31.323730342 +0000 UTC m=+2209.186436251" Oct 30 09:31:32.105869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:32.105783 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g"] Oct 30 09:31:32.110067 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:32.110044 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" Oct 30 09:31:32.138709 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:32.138671 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g"] Oct 30 09:31:32.154444 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:32.154400 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf218ba-ee3e-44ba-a0e1-676804744021-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d3c9f02-tvf7g\" (UID: \"adf218ba-ee3e-44ba-a0e1-676804744021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" Oct 30 09:31:32.154704 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:32.154460 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf218ba-ee3e-44ba-a0e1-676804744021-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d3c9f02-tvf7g\" (UID: \"adf218ba-ee3e-44ba-a0e1-676804744021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" Oct 30 09:31:32.154771 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:32.154715 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjrb5\" (UniqueName: \"kubernetes.io/projected/adf218ba-ee3e-44ba-a0e1-676804744021-kube-api-access-tjrb5\") pod \"maas-api-revoke-sub-8d3c9f02-tvf7g\" (UID: \"adf218ba-ee3e-44ba-a0e1-676804744021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" Oct 30 09:31:32.255767 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:32.255721 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrb5\" (UniqueName: \"kubernetes.io/projected/adf218ba-ee3e-44ba-a0e1-676804744021-kube-api-access-tjrb5\") pod \"maas-api-revoke-sub-8d3c9f02-tvf7g\" (UID: \"adf218ba-ee3e-44ba-a0e1-676804744021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" Oct 30 09:31:32.256042 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:32.255799 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf218ba-ee3e-44ba-a0e1-676804744021-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d3c9f02-tvf7g\" (UID: \"adf218ba-ee3e-44ba-a0e1-676804744021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" Oct 30 09:31:32.256042 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:32.255835 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf218ba-ee3e-44ba-a0e1-676804744021-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d3c9f02-tvf7g\" (UID: \"adf218ba-ee3e-44ba-a0e1-676804744021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" Oct 30 09:31:32.256861 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:32.256832 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf218ba-ee3e-44ba-a0e1-676804744021-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d3c9f02-tvf7g\" (UID: \"adf218ba-ee3e-44ba-a0e1-676804744021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" Oct 30 09:31:32.258448 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:32.258424 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf218ba-ee3e-44ba-a0e1-676804744021-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d3c9f02-tvf7g\" (UID: \"adf218ba-ee3e-44ba-a0e1-676804744021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" Oct 30 09:31:32.277755 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:32.277730 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrb5\" (UniqueName: \"kubernetes.io/projected/adf218ba-ee3e-44ba-a0e1-676804744021-kube-api-access-tjrb5\") pod \"maas-api-revoke-sub-8d3c9f02-tvf7g\" (UID: \"adf218ba-ee3e-44ba-a0e1-676804744021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" Oct 30 09:31:32.284798 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:32.284773 5938 generic.go:356] "Generic (PLEG): container finished" podID="79dc7829-7dcb-4118-a759-ed05d381e749" containerID="71a900597b1f0d399daf7c1768542207ec21a3d8a0dc10cce627dbfbdbe4afb4" exitCode=0 Oct 30 09:31:32.284915 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:32.284847 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" event={"ID":"79dc7829-7dcb-4118-a759-ed05d381e749","Type":"ContainerDied","Data":"71a900597b1f0d399daf7c1768542207ec21a3d8a0dc10cce627dbfbdbe4afb4"} Oct 30 09:31:32.428681 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:32.428647 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" Oct 30 09:31:32.746901 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:32.746653 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g"] Oct 30 09:31:32.749526 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:31:32.748741 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf218ba_ee3e_44ba_a0e1_676804744021.slice/crio-d205cdb2f0908e0a436cddff5d5e1d63efd4beae304602b7c74b39e87f37313c WatchSource:0}: Error finding container d205cdb2f0908e0a436cddff5d5e1d63efd4beae304602b7c74b39e87f37313c: Status 404 returned error can't find the container with id d205cdb2f0908e0a436cddff5d5e1d63efd4beae304602b7c74b39e87f37313c Oct 30 09:31:33.295053 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:33.294927 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" event={"ID":"adf218ba-ee3e-44ba-a0e1-676804744021","Type":"ContainerStarted","Data":"0d4475200c7082991b306f2be29de6d210704ac2495f1841237d8695889a0ce1"} Oct 30 09:31:33.295053 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:33.294998 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" event={"ID":"adf218ba-ee3e-44ba-a0e1-676804744021","Type":"ContainerStarted","Data":"d205cdb2f0908e0a436cddff5d5e1d63efd4beae304602b7c74b39e87f37313c"} Oct 30 09:31:33.456427 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:33.456403 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" Oct 30 09:31:33.482362 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:33.482333 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79dc7829-7dcb-4118-a759-ed05d381e749-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79dc7829-7dcb-4118-a759-ed05d381e749-maas-api-openshift-ca\") pod \"79dc7829-7dcb-4118-a759-ed05d381e749\" (UID: \"79dc7829-7dcb-4118-a759-ed05d381e749\") " Oct 30 09:31:33.482530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:33.482397 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79dc7829-7dcb-4118-a759-ed05d381e749-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79dc7829-7dcb-4118-a759-ed05d381e749-maas-api-xks-ca\") pod \"79dc7829-7dcb-4118-a759-ed05d381e749\" (UID: \"79dc7829-7dcb-4118-a759-ed05d381e749\") " Oct 30 09:31:33.482530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:33.482526 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79dc7829-7dcb-4118-a759-ed05d381e749-kube-api-access-9rdwg\" (UniqueName: \"kubernetes.io/projected/79dc7829-7dcb-4118-a759-ed05d381e749-kube-api-access-9rdwg\") pod \"79dc7829-7dcb-4118-a759-ed05d381e749\" (UID: \"79dc7829-7dcb-4118-a759-ed05d381e749\") " Oct 30 09:31:33.482778 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:33.482755 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79dc7829-7dcb-4118-a759-ed05d381e749-maas-api-openshift-ca" pod "79dc7829-7dcb-4118-a759-ed05d381e749" (UID: "79dc7829-7dcb-4118-a759-ed05d381e749"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:33.483120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:33.483054 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79dc7829-7dcb-4118-a759-ed05d381e749-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:33.484804 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:33.484779 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79dc7829-7dcb-4118-a759-ed05d381e749-maas-api-xks-ca" pod "79dc7829-7dcb-4118-a759-ed05d381e749" (UID: "79dc7829-7dcb-4118-a759-ed05d381e749"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:31:33.484902 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:33.484803 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79dc7829-7dcb-4118-a759-ed05d381e749-kube-api-access-9rdwg" pod "79dc7829-7dcb-4118-a759-ed05d381e749" (UID: "79dc7829-7dcb-4118-a759-ed05d381e749"). InnerVolumeSpecName "kube-api-access-9rdwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:33.505787 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:33.505742 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" podStartSLOduration=1.505727259 podStartE2EDuration="1.505727259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:31:33.340951161 +0000 UTC m=+2211.203656981" watchObservedRunningTime="2026-10-30 09:31:33.505727259 +0000 UTC m=+2211.368433079" Oct 30 09:31:33.584522 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:33.584443 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rdwg\" (UniqueName: \"kubernetes.io/projected/79dc7829-7dcb-4118-a759-ed05d381e749-kube-api-access-9rdwg\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:33.584522 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:33.584473 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79dc7829-7dcb-4118-a759-ed05d381e749-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:34.303080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:34.303048 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" Oct 30 09:31:34.303080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:34.303073 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm" event={"ID":"79dc7829-7dcb-4118-a759-ed05d381e749","Type":"ContainerDied","Data":"8f255dff05ee03bfd5be7bc5bc54aa5b39a365114f06474e38251bf6d9081552"} Oct 30 09:31:34.303657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:34.303122 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f255dff05ee03bfd5be7bc5bc54aa5b39a365114f06474e38251bf6d9081552" Oct 30 09:31:34.306319 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:34.306287 5938 generic.go:356] "Generic (PLEG): container finished" podID="adf218ba-ee3e-44ba-a0e1-676804744021" containerID="0d4475200c7082991b306f2be29de6d210704ac2495f1841237d8695889a0ce1" exitCode=0 Oct 30 09:31:34.306453 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:34.306341 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" event={"ID":"adf218ba-ee3e-44ba-a0e1-676804744021","Type":"ContainerDied","Data":"0d4475200c7082991b306f2be29de6d210704ac2495f1841237d8695889a0ce1"} Oct 30 09:31:35.471441 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:35.471408 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" Oct 30 09:31:35.511561 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:35.511526 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf218ba-ee3e-44ba-a0e1-676804744021-kube-api-access-tjrb5\" (UniqueName: \"kubernetes.io/projected/adf218ba-ee3e-44ba-a0e1-676804744021-kube-api-access-tjrb5\") pod \"adf218ba-ee3e-44ba-a0e1-676804744021\" (UID: \"adf218ba-ee3e-44ba-a0e1-676804744021\") " Oct 30 09:31:35.511819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:35.511576 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adf218ba-ee3e-44ba-a0e1-676804744021-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf218ba-ee3e-44ba-a0e1-676804744021-maas-api-xks-ca\") pod \"adf218ba-ee3e-44ba-a0e1-676804744021\" (UID: \"adf218ba-ee3e-44ba-a0e1-676804744021\") " Oct 30 09:31:35.511819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:35.511656 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adf218ba-ee3e-44ba-a0e1-676804744021-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf218ba-ee3e-44ba-a0e1-676804744021-maas-api-openshift-ca\") pod \"adf218ba-ee3e-44ba-a0e1-676804744021\" (UID: \"adf218ba-ee3e-44ba-a0e1-676804744021\") " Oct 30 09:31:35.512275 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:35.512156 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf218ba-ee3e-44ba-a0e1-676804744021-maas-api-openshift-ca" pod "adf218ba-ee3e-44ba-a0e1-676804744021" (UID: "adf218ba-ee3e-44ba-a0e1-676804744021"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:35.512275 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:35.512333 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf218ba-ee3e-44ba-a0e1-676804744021-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:35.514542 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:35.514508 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adf218ba-ee3e-44ba-a0e1-676804744021-maas-api-xks-ca" pod "adf218ba-ee3e-44ba-a0e1-676804744021" (UID: "adf218ba-ee3e-44ba-a0e1-676804744021"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:31:35.514699 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:35.514647 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf218ba-ee3e-44ba-a0e1-676804744021-kube-api-access-tjrb5" pod "adf218ba-ee3e-44ba-a0e1-676804744021" (UID: "adf218ba-ee3e-44ba-a0e1-676804744021"). InnerVolumeSpecName "kube-api-access-tjrb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:35.613447 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:35.613361 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjrb5\" (UniqueName: \"kubernetes.io/projected/adf218ba-ee3e-44ba-a0e1-676804744021-kube-api-access-tjrb5\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:35.613447 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:35.613392 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf218ba-ee3e-44ba-a0e1-676804744021-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:36.320922 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:36.320893 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" Oct 30 09:31:36.321106 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:36.320930 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g" event={"ID":"adf218ba-ee3e-44ba-a0e1-676804744021","Type":"ContainerDied","Data":"d205cdb2f0908e0a436cddff5d5e1d63efd4beae304602b7c74b39e87f37313c"} Oct 30 09:31:36.321106 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:36.320978 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d205cdb2f0908e0a436cddff5d5e1d63efd4beae304602b7c74b39e87f37313c" Oct 30 09:31:40.530096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.530059 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v"] Oct 30 09:31:40.530974 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.530937 5938 cpu_manager.go:414] "containerMap: removing container" podUID="79dc7829-7dcb-4118-a759-ed05d381e749" containerName="revoke-keys" Oct 30 09:31:40.530974 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.530965 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79dc7829-7dcb-4118-a759-ed05d381e749" containerName="revoke-keys" Oct 30 09:31:40.531191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.530986 5938 cpu_manager.go:414] "containerMap: removing container" podUID="adf218ba-ee3e-44ba-a0e1-676804744021" containerName="revoke-keys" Oct 30 09:31:40.531191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.530995 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adf218ba-ee3e-44ba-a0e1-676804744021" containerName="revoke-keys" Oct 30 09:31:40.531191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.531187 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="79dc7829-7dcb-4118-a759-ed05d381e749" containerName="revoke-keys" Oct 30 09:31:40.531411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.531199 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="adf218ba-ee3e-44ba-a0e1-676804744021" containerName="revoke-keys" Oct 30 09:31:40.535468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.535442 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" Oct 30 09:31:40.555137 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.555108 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v"] Oct 30 09:31:40.565388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.564787 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:31:40.565388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.564821 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:31:40.682638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.682605 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab08dc6e-1b67-4d88-83d6-24edf1376127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264868e2-99f8v\" (UID: \"ab08dc6e-1b67-4d88-83d6-24edf1376127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" Oct 30 09:31:40.682828 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.682807 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtgqp\" (UniqueName: \"kubernetes.io/projected/ab08dc6e-1b67-4d88-83d6-24edf1376127-kube-api-access-xtgqp\") pod \"maas-api-revoke-sub-264868e2-99f8v\" (UID: \"ab08dc6e-1b67-4d88-83d6-24edf1376127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" Oct 30 09:31:40.682876 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.682856 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab08dc6e-1b67-4d88-83d6-24edf1376127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264868e2-99f8v\" (UID: \"ab08dc6e-1b67-4d88-83d6-24edf1376127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" Oct 30 09:31:40.785665 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.784231 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab08dc6e-1b67-4d88-83d6-24edf1376127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264868e2-99f8v\" (UID: \"ab08dc6e-1b67-4d88-83d6-24edf1376127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" Oct 30 09:31:40.785665 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.784424 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtgqp\" (UniqueName: \"kubernetes.io/projected/ab08dc6e-1b67-4d88-83d6-24edf1376127-kube-api-access-xtgqp\") pod \"maas-api-revoke-sub-264868e2-99f8v\" (UID: \"ab08dc6e-1b67-4d88-83d6-24edf1376127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" Oct 30 09:31:40.785665 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.784460 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab08dc6e-1b67-4d88-83d6-24edf1376127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264868e2-99f8v\" (UID: \"ab08dc6e-1b67-4d88-83d6-24edf1376127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" Oct 30 09:31:40.787631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.786337 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab08dc6e-1b67-4d88-83d6-24edf1376127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264868e2-99f8v\" (UID: \"ab08dc6e-1b67-4d88-83d6-24edf1376127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" Oct 30 09:31:40.790346 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.788338 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab08dc6e-1b67-4d88-83d6-24edf1376127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264868e2-99f8v\" (UID: \"ab08dc6e-1b67-4d88-83d6-24edf1376127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" Oct 30 09:31:40.808606 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.808552 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtgqp\" (UniqueName: \"kubernetes.io/projected/ab08dc6e-1b67-4d88-83d6-24edf1376127-kube-api-access-xtgqp\") pod \"maas-api-revoke-sub-264868e2-99f8v\" (UID: \"ab08dc6e-1b67-4d88-83d6-24edf1376127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" Oct 30 09:31:40.856296 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:40.856251 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" Oct 30 09:31:41.208614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:41.207833 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v"] Oct 30 09:31:41.458613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:41.444390 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" event={"ID":"ab08dc6e-1b67-4d88-83d6-24edf1376127","Type":"ContainerStarted","Data":"31383d560471271b9cb55b4a42e7427ea65a768d81aeac0d094c1b3a8cf34896"} Oct 30 09:31:42.394614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:42.393736 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:42.455739 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:42.455691 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 09:31:42.499998 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:42.499957 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" event={"ID":"ab08dc6e-1b67-4d88-83d6-24edf1376127","Type":"ContainerStarted","Data":"f5b0426e20b0c8162b57643237c549ab23f7fc93290901bb037ee15d6417792e"} Oct 30 09:31:42.548388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:42.548338 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:42.613617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:42.586849 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" podStartSLOduration=2.586828306 podStartE2EDuration="2.586828306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:31:42.564206176 +0000 UTC m=+2220.426911999" watchObservedRunningTime="2026-10-30 09:31:42.586828306 +0000 UTC m=+2220.449534129" Oct 30 09:31:42.636708 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:42.616386 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:42.637776 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:42.592491 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:42.840624 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:42.839729 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:43.515198 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:43.515100 5938 generic.go:356] "Generic (PLEG): container finished" podID="ab08dc6e-1b67-4d88-83d6-24edf1376127" containerID="f5b0426e20b0c8162b57643237c549ab23f7fc93290901bb037ee15d6417792e" exitCode=0 Oct 30 09:31:43.515744 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:43.515373 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" event={"ID":"ab08dc6e-1b67-4d88-83d6-24edf1376127","Type":"ContainerDied","Data":"f5b0426e20b0c8162b57643237c549ab23f7fc93290901bb037ee15d6417792e"} Oct 30 09:31:44.717280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:44.717243 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" Oct 30 09:31:44.891523 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:44.891486 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab08dc6e-1b67-4d88-83d6-24edf1376127-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab08dc6e-1b67-4d88-83d6-24edf1376127-maas-api-xks-ca\") pod \"ab08dc6e-1b67-4d88-83d6-24edf1376127\" (UID: \"ab08dc6e-1b67-4d88-83d6-24edf1376127\") " Oct 30 09:31:44.891748 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:44.891669 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab08dc6e-1b67-4d88-83d6-24edf1376127-kube-api-access-xtgqp\" (UniqueName: \"kubernetes.io/projected/ab08dc6e-1b67-4d88-83d6-24edf1376127-kube-api-access-xtgqp\") pod \"ab08dc6e-1b67-4d88-83d6-24edf1376127\" (UID: \"ab08dc6e-1b67-4d88-83d6-24edf1376127\") " Oct 30 09:31:44.891748 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:44.891709 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab08dc6e-1b67-4d88-83d6-24edf1376127-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab08dc6e-1b67-4d88-83d6-24edf1376127-maas-api-openshift-ca\") pod \"ab08dc6e-1b67-4d88-83d6-24edf1376127\" (UID: \"ab08dc6e-1b67-4d88-83d6-24edf1376127\") " Oct 30 09:31:44.892146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:44.892117 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab08dc6e-1b67-4d88-83d6-24edf1376127-maas-api-openshift-ca" pod "ab08dc6e-1b67-4d88-83d6-24edf1376127" (UID: "ab08dc6e-1b67-4d88-83d6-24edf1376127"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:44.893997 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:44.893970 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab08dc6e-1b67-4d88-83d6-24edf1376127-maas-api-xks-ca" pod "ab08dc6e-1b67-4d88-83d6-24edf1376127" (UID: "ab08dc6e-1b67-4d88-83d6-24edf1376127"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:31:44.894152 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:44.894100 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab08dc6e-1b67-4d88-83d6-24edf1376127-kube-api-access-xtgqp" pod "ab08dc6e-1b67-4d88-83d6-24edf1376127" (UID: "ab08dc6e-1b67-4d88-83d6-24edf1376127"). InnerVolumeSpecName "kube-api-access-xtgqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:44.993431 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:44.993387 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtgqp\" (UniqueName: \"kubernetes.io/projected/ab08dc6e-1b67-4d88-83d6-24edf1376127-kube-api-access-xtgqp\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:44.993431 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:44.993426 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab08dc6e-1b67-4d88-83d6-24edf1376127-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:44.993846 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:44.993441 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab08dc6e-1b67-4d88-83d6-24edf1376127-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:45.530812 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:45.530775 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" Oct 30 09:31:45.530812 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:45.530782 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v" event={"ID":"ab08dc6e-1b67-4d88-83d6-24edf1376127","Type":"ContainerDied","Data":"31383d560471271b9cb55b4a42e7427ea65a768d81aeac0d094c1b3a8cf34896"} Oct 30 09:31:45.531044 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:45.530829 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31383d560471271b9cb55b4a42e7427ea65a768d81aeac0d094c1b3a8cf34896" Oct 30 09:31:45.801442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:45.801330 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx"] Oct 30 09:31:45.807054 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:45.807025 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d95d35-95gfx"] Oct 30 09:31:45.814025 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:31:45.813991 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:31:46.257817 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:46.257780 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37e62be0-a6d4-411b-a7ac-b91b066f24ec" path="/var/lib/kubelet/pods/37e62be0-a6d4-411b-a7ac-b91b066f24ec/volumes" Oct 30 09:31:47.224747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.224709 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7"] Oct 30 09:31:47.225314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.225297 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ab08dc6e-1b67-4d88-83d6-24edf1376127" containerName="revoke-keys" Oct 30 09:31:47.225360 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.225316 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab08dc6e-1b67-4d88-83d6-24edf1376127" containerName="revoke-keys" Oct 30 09:31:47.225454 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.225444 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab08dc6e-1b67-4d88-83d6-24edf1376127" containerName="revoke-keys" Oct 30 09:31:47.230202 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.230180 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" Oct 30 09:31:47.252647 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.252611 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7"] Oct 30 09:31:47.261393 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.261361 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:31:47.261567 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.261409 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:31:47.325623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.325555 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8931547c-3aea-456f-8eef-27f96da59180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb21f57c-jccz7\" (UID: \"8931547c-3aea-456f-8eef-27f96da59180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" Oct 30 09:31:47.325812 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.325696 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9mz2\" (UniqueName: \"kubernetes.io/projected/8931547c-3aea-456f-8eef-27f96da59180-kube-api-access-n9mz2\") pod \"maas-api-revoke-sub-eb21f57c-jccz7\" (UID: \"8931547c-3aea-456f-8eef-27f96da59180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" Oct 30 09:31:47.325812 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.325766 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8931547c-3aea-456f-8eef-27f96da59180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb21f57c-jccz7\" (UID: \"8931547c-3aea-456f-8eef-27f96da59180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" Oct 30 09:31:47.427000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.426954 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8931547c-3aea-456f-8eef-27f96da59180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb21f57c-jccz7\" (UID: \"8931547c-3aea-456f-8eef-27f96da59180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" Oct 30 09:31:47.427256 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.427227 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8931547c-3aea-456f-8eef-27f96da59180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb21f57c-jccz7\" (UID: \"8931547c-3aea-456f-8eef-27f96da59180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" Oct 30 09:31:47.427350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.427311 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9mz2\" (UniqueName: \"kubernetes.io/projected/8931547c-3aea-456f-8eef-27f96da59180-kube-api-access-n9mz2\") pod \"maas-api-revoke-sub-eb21f57c-jccz7\" (UID: \"8931547c-3aea-456f-8eef-27f96da59180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" Oct 30 09:31:47.427977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.427948 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8931547c-3aea-456f-8eef-27f96da59180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb21f57c-jccz7\" (UID: \"8931547c-3aea-456f-8eef-27f96da59180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" Oct 30 09:31:47.429565 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.429543 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8931547c-3aea-456f-8eef-27f96da59180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb21f57c-jccz7\" (UID: \"8931547c-3aea-456f-8eef-27f96da59180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" Oct 30 09:31:47.459450 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.459415 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9mz2\" (UniqueName: \"kubernetes.io/projected/8931547c-3aea-456f-8eef-27f96da59180-kube-api-access-n9mz2\") pod \"maas-api-revoke-sub-eb21f57c-jccz7\" (UID: \"8931547c-3aea-456f-8eef-27f96da59180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" Oct 30 09:31:47.544756 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.544671 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" Oct 30 09:31:47.851260 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:47.851232 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7"] Oct 30 09:31:47.852532 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:31:47.852505 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8931547c_3aea_456f_8eef_27f96da59180.slice/crio-541cdfef0c338272129df363596b02e6b63f66b459eb137010bb73095976425c WatchSource:0}: Error finding container 541cdfef0c338272129df363596b02e6b63f66b459eb137010bb73095976425c: Status 404 returned error can't find the container with id 541cdfef0c338272129df363596b02e6b63f66b459eb137010bb73095976425c Oct 30 09:31:48.555526 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:48.555422 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" event={"ID":"8931547c-3aea-456f-8eef-27f96da59180","Type":"ContainerStarted","Data":"b15d5261143893850b67704e268c190af929eff9e2af163438325eea10cf1800"} Oct 30 09:31:48.555526 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:48.555497 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" event={"ID":"8931547c-3aea-456f-8eef-27f96da59180","Type":"ContainerStarted","Data":"541cdfef0c338272129df363596b02e6b63f66b459eb137010bb73095976425c"} Oct 30 09:31:48.596889 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:48.596833 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" podStartSLOduration=1.596817759 podStartE2EDuration="1.596817759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:31:48.596388709 +0000 UTC m=+2226.459094533" watchObservedRunningTime="2026-10-30 09:31:48.596817759 +0000 UTC m=+2226.459523577" Oct 30 09:31:49.563933 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:49.563900 5938 generic.go:356] "Generic (PLEG): container finished" podID="8931547c-3aea-456f-8eef-27f96da59180" containerID="b15d5261143893850b67704e268c190af929eff9e2af163438325eea10cf1800" exitCode=0 Oct 30 09:31:49.564357 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:49.563941 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" event={"ID":"8931547c-3aea-456f-8eef-27f96da59180","Type":"ContainerDied","Data":"b15d5261143893850b67704e268c190af929eff9e2af163438325eea10cf1800"} Oct 30 09:31:50.749165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:50.749133 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" Oct 30 09:31:50.878905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:50.878874 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8931547c-3aea-456f-8eef-27f96da59180-kube-api-access-n9mz2\" (UniqueName: \"kubernetes.io/projected/8931547c-3aea-456f-8eef-27f96da59180-kube-api-access-n9mz2\") pod \"8931547c-3aea-456f-8eef-27f96da59180\" (UID: \"8931547c-3aea-456f-8eef-27f96da59180\") " Oct 30 09:31:50.879107 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:50.878959 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8931547c-3aea-456f-8eef-27f96da59180-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8931547c-3aea-456f-8eef-27f96da59180-maas-api-openshift-ca\") pod \"8931547c-3aea-456f-8eef-27f96da59180\" (UID: \"8931547c-3aea-456f-8eef-27f96da59180\") " Oct 30 09:31:50.879107 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:50.879046 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8931547c-3aea-456f-8eef-27f96da59180-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8931547c-3aea-456f-8eef-27f96da59180-maas-api-xks-ca\") pod \"8931547c-3aea-456f-8eef-27f96da59180\" (UID: \"8931547c-3aea-456f-8eef-27f96da59180\") " Oct 30 09:31:50.879341 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:50.879315 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8931547c-3aea-456f-8eef-27f96da59180-maas-api-openshift-ca" pod "8931547c-3aea-456f-8eef-27f96da59180" (UID: "8931547c-3aea-456f-8eef-27f96da59180"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:50.879539 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:50.879522 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8931547c-3aea-456f-8eef-27f96da59180-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:50.881264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:50.881238 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8931547c-3aea-456f-8eef-27f96da59180-maas-api-xks-ca" pod "8931547c-3aea-456f-8eef-27f96da59180" (UID: "8931547c-3aea-456f-8eef-27f96da59180"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:31:50.881373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:50.881272 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8931547c-3aea-456f-8eef-27f96da59180-kube-api-access-n9mz2" pod "8931547c-3aea-456f-8eef-27f96da59180" (UID: "8931547c-3aea-456f-8eef-27f96da59180"). InnerVolumeSpecName "kube-api-access-n9mz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:50.980898 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:50.980864 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9mz2\" (UniqueName: \"kubernetes.io/projected/8931547c-3aea-456f-8eef-27f96da59180-kube-api-access-n9mz2\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:50.980898 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:50.980895 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8931547c-3aea-456f-8eef-27f96da59180-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:51.584774 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:51.584735 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" event={"ID":"8931547c-3aea-456f-8eef-27f96da59180","Type":"ContainerDied","Data":"541cdfef0c338272129df363596b02e6b63f66b459eb137010bb73095976425c"} Oct 30 09:31:51.585024 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:51.585005 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="541cdfef0c338272129df363596b02e6b63f66b459eb137010bb73095976425c" Oct 30 09:31:51.585168 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:51.584768 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7" Oct 30 09:31:55.300319 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.300286 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r"] Oct 30 09:31:55.300933 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.300917 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8931547c-3aea-456f-8eef-27f96da59180" containerName="revoke-keys" Oct 30 09:31:55.300988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.300935 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8931547c-3aea-456f-8eef-27f96da59180" containerName="revoke-keys" Oct 30 09:31:55.301028 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.301024 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8931547c-3aea-456f-8eef-27f96da59180" containerName="revoke-keys" Oct 30 09:31:55.304750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.304729 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" Oct 30 09:31:55.324207 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.324166 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r"] Oct 30 09:31:55.332862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.332829 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:31:55.333024 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.332902 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:31:55.453164 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.453127 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1227bf8a-0716-48eb-9530-86ae66f12acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e5099a-hh84r\" (UID: \"1227bf8a-0716-48eb-9530-86ae66f12acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" Oct 30 09:31:55.453164 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.453173 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1227bf8a-0716-48eb-9530-86ae66f12acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e5099a-hh84r\" (UID: \"1227bf8a-0716-48eb-9530-86ae66f12acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" Oct 30 09:31:55.453480 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.453297 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx958\" (UniqueName: \"kubernetes.io/projected/1227bf8a-0716-48eb-9530-86ae66f12acd-kube-api-access-fx958\") pod \"maas-api-revoke-sub-96e5099a-hh84r\" (UID: \"1227bf8a-0716-48eb-9530-86ae66f12acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" Oct 30 09:31:55.555244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.555148 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1227bf8a-0716-48eb-9530-86ae66f12acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e5099a-hh84r\" (UID: \"1227bf8a-0716-48eb-9530-86ae66f12acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" Oct 30 09:31:55.555244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.555210 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1227bf8a-0716-48eb-9530-86ae66f12acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e5099a-hh84r\" (UID: \"1227bf8a-0716-48eb-9530-86ae66f12acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" Oct 30 09:31:55.555490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.555295 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx958\" (UniqueName: \"kubernetes.io/projected/1227bf8a-0716-48eb-9530-86ae66f12acd-kube-api-access-fx958\") pod \"maas-api-revoke-sub-96e5099a-hh84r\" (UID: \"1227bf8a-0716-48eb-9530-86ae66f12acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" Oct 30 09:31:55.555929 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.555904 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1227bf8a-0716-48eb-9530-86ae66f12acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e5099a-hh84r\" (UID: \"1227bf8a-0716-48eb-9530-86ae66f12acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" Oct 30 09:31:55.557837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.557809 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1227bf8a-0716-48eb-9530-86ae66f12acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e5099a-hh84r\" (UID: \"1227bf8a-0716-48eb-9530-86ae66f12acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" Oct 30 09:31:55.577117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.577081 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx958\" (UniqueName: \"kubernetes.io/projected/1227bf8a-0716-48eb-9530-86ae66f12acd-kube-api-access-fx958\") pod \"maas-api-revoke-sub-96e5099a-hh84r\" (UID: \"1227bf8a-0716-48eb-9530-86ae66f12acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" Oct 30 09:31:55.632615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:55.627934 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" Oct 30 09:31:56.040285 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:56.040249 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r"] Oct 30 09:31:56.040436 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:31:56.040276 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1227bf8a_0716_48eb_9530_86ae66f12acd.slice/crio-e6ce77d2bb270181ced502c58a979ca6f736be7247c1b28f14edadf1a591d6a5 WatchSource:0}: Error finding container e6ce77d2bb270181ced502c58a979ca6f736be7247c1b28f14edadf1a591d6a5: Status 404 returned error can't find the container with id e6ce77d2bb270181ced502c58a979ca6f736be7247c1b28f14edadf1a591d6a5 Oct 30 09:31:56.641109 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:56.641073 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" event={"ID":"1227bf8a-0716-48eb-9530-86ae66f12acd","Type":"ContainerStarted","Data":"5a5c30b15822b2444aaa795dfe5e042f7be3b7588055b0917f2a4de15df4114f"} Oct 30 09:31:56.641580 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:56.641135 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" event={"ID":"1227bf8a-0716-48eb-9530-86ae66f12acd","Type":"ContainerStarted","Data":"e6ce77d2bb270181ced502c58a979ca6f736be7247c1b28f14edadf1a591d6a5"} Oct 30 09:31:56.683601 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:56.683526 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" podStartSLOduration=1.683510403 podStartE2EDuration="1.683510403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:31:56.680719114 +0000 UTC m=+2234.543424935" watchObservedRunningTime="2026-10-30 09:31:56.683510403 +0000 UTC m=+2234.546216223" Oct 30 09:31:57.649727 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:57.649693 5938 generic.go:356] "Generic (PLEG): container finished" podID="1227bf8a-0716-48eb-9530-86ae66f12acd" containerID="5a5c30b15822b2444aaa795dfe5e042f7be3b7588055b0917f2a4de15df4114f" exitCode=0 Oct 30 09:31:57.650151 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:57.649739 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" event={"ID":"1227bf8a-0716-48eb-9530-86ae66f12acd","Type":"ContainerDied","Data":"5a5c30b15822b2444aaa795dfe5e042f7be3b7588055b0917f2a4de15df4114f"} Oct 30 09:31:58.812193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:58.812163 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" Oct 30 09:31:58.907467 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:58.907377 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1227bf8a-0716-48eb-9530-86ae66f12acd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1227bf8a-0716-48eb-9530-86ae66f12acd-maas-api-xks-ca\") pod \"1227bf8a-0716-48eb-9530-86ae66f12acd\" (UID: \"1227bf8a-0716-48eb-9530-86ae66f12acd\") " Oct 30 09:31:58.907467 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:58.907460 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1227bf8a-0716-48eb-9530-86ae66f12acd-kube-api-access-fx958\" (UniqueName: \"kubernetes.io/projected/1227bf8a-0716-48eb-9530-86ae66f12acd-kube-api-access-fx958\") pod \"1227bf8a-0716-48eb-9530-86ae66f12acd\" (UID: \"1227bf8a-0716-48eb-9530-86ae66f12acd\") " Oct 30 09:31:58.907761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:58.907544 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1227bf8a-0716-48eb-9530-86ae66f12acd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1227bf8a-0716-48eb-9530-86ae66f12acd-maas-api-openshift-ca\") pod \"1227bf8a-0716-48eb-9530-86ae66f12acd\" (UID: \"1227bf8a-0716-48eb-9530-86ae66f12acd\") " Oct 30 09:31:58.907962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:58.907929 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1227bf8a-0716-48eb-9530-86ae66f12acd-maas-api-openshift-ca" pod "1227bf8a-0716-48eb-9530-86ae66f12acd" (UID: "1227bf8a-0716-48eb-9530-86ae66f12acd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:58.908271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:58.908253 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1227bf8a-0716-48eb-9530-86ae66f12acd-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:58.909811 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:58.909787 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1227bf8a-0716-48eb-9530-86ae66f12acd-kube-api-access-fx958" pod "1227bf8a-0716-48eb-9530-86ae66f12acd" (UID: "1227bf8a-0716-48eb-9530-86ae66f12acd"). InnerVolumeSpecName "kube-api-access-fx958". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:58.909914 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:58.909841 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1227bf8a-0716-48eb-9530-86ae66f12acd-maas-api-xks-ca" pod "1227bf8a-0716-48eb-9530-86ae66f12acd" (UID: "1227bf8a-0716-48eb-9530-86ae66f12acd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:31:59.009267 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:59.009230 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1227bf8a-0716-48eb-9530-86ae66f12acd-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:59.009267 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:59.009264 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx958\" (UniqueName: \"kubernetes.io/projected/1227bf8a-0716-48eb-9530-86ae66f12acd-kube-api-access-fx958\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:31:59.666545 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:59.666410 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" Oct 30 09:31:59.666545 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:59.666410 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r" event={"ID":"1227bf8a-0716-48eb-9530-86ae66f12acd","Type":"ContainerDied","Data":"e6ce77d2bb270181ced502c58a979ca6f736be7247c1b28f14edadf1a591d6a5"} Oct 30 09:31:59.666545 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:31:59.666561 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6ce77d2bb270181ced502c58a979ca6f736be7247c1b28f14edadf1a591d6a5" Oct 30 09:32:01.824851 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:01.824821 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf"] Oct 30 09:32:01.825396 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:01.825382 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1227bf8a-0716-48eb-9530-86ae66f12acd" containerName="revoke-keys" Oct 30 09:32:01.825445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:01.825398 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1227bf8a-0716-48eb-9530-86ae66f12acd" containerName="revoke-keys" Oct 30 09:32:01.825541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:01.825530 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1227bf8a-0716-48eb-9530-86ae66f12acd" containerName="revoke-keys" Oct 30 09:32:01.829225 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:01.829206 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf" Oct 30 09:32:01.849917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:01.849892 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf"] Oct 30 09:32:01.901504 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:01.901476 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:32:01.901504 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:01.901497 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:32:01.968368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:01.968321 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-705a1a6c-pzrwf\" (UID: \"4f98c0a5-33ff-4e3b-946d-1b50bc80502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf" Oct 30 09:32:01.968580 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:01.968439 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwxd2\" (UniqueName: \"kubernetes.io/projected/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-kube-api-access-kwxd2\") pod \"maas-api-revoke-sub-705a1a6c-pzrwf\" (UID: \"4f98c0a5-33ff-4e3b-946d-1b50bc80502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf" Oct 30 09:32:01.968580 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:01.968485 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-705a1a6c-pzrwf\" (UID: \"4f98c0a5-33ff-4e3b-946d-1b50bc80502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf" Oct 30 09:32:02.069768 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:02.069736 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-705a1a6c-pzrwf\" (UID: \"4f98c0a5-33ff-4e3b-946d-1b50bc80502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf" Oct 30 09:32:02.069942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:02.069800 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwxd2\" (UniqueName: \"kubernetes.io/projected/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-kube-api-access-kwxd2\") pod \"maas-api-revoke-sub-705a1a6c-pzrwf\" (UID: \"4f98c0a5-33ff-4e3b-946d-1b50bc80502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf" Oct 30 09:32:02.069942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:02.069834 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-705a1a6c-pzrwf\" (UID: \"4f98c0a5-33ff-4e3b-946d-1b50bc80502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf" Oct 30 09:32:02.070632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:02.070572 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-705a1a6c-pzrwf\" (UID: \"4f98c0a5-33ff-4e3b-946d-1b50bc80502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf" Oct 30 09:32:02.072410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:02.072391 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-705a1a6c-pzrwf\" (UID: \"4f98c0a5-33ff-4e3b-946d-1b50bc80502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf" Oct 30 09:32:02.091901 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:02.091840 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwxd2\" (UniqueName: \"kubernetes.io/projected/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-kube-api-access-kwxd2\") pod \"maas-api-revoke-sub-705a1a6c-pzrwf\" (UID: \"4f98c0a5-33ff-4e3b-946d-1b50bc80502b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf" Oct 30 09:32:02.144512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:02.144479 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf" Oct 30 09:32:02.454332 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:02.454301 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf"] Oct 30 09:32:02.454864 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:32:02.454829 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f98c0a5_33ff_4e3b_946d_1b50bc80502b.slice/crio-b1c0446a9b871b30a48f76c2f3e2a0be1560877615437fd7a21cd75ae09d30ab WatchSource:0}: Error finding container b1c0446a9b871b30a48f76c2f3e2a0be1560877615437fd7a21cd75ae09d30ab: Status 404 returned error can't find the container with id b1c0446a9b871b30a48f76c2f3e2a0be1560877615437fd7a21cd75ae09d30ab Oct 30 09:32:02.688019 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:02.687982 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf" event={"ID":"4f98c0a5-33ff-4e3b-946d-1b50bc80502b","Type":"ContainerStarted","Data":"b1c0446a9b871b30a48f76c2f3e2a0be1560877615437fd7a21cd75ae09d30ab"} Oct 30 09:32:03.697097 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:03.697059 5938 generic.go:356] "Generic (PLEG): container finished" podID="4f98c0a5-33ff-4e3b-946d-1b50bc80502b" containerID="bc8438225611422621417b01e4835794948c2095448a7b413ae0da25b3f8a910" exitCode=0 Oct 30 09:32:03.697489 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:03.697136 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf" event={"ID":"4f98c0a5-33ff-4e3b-946d-1b50bc80502b","Type":"ContainerDied","Data":"bc8438225611422621417b01e4835794948c2095448a7b413ae0da25b3f8a910"} Oct 30 09:32:03.713265 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:03.713235 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g"] Oct 30 09:32:03.713535 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:03.713507 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" podUID="0daeb10f-9b40-4ecb-bccd-d03c06be57c7" containerName="discovery" containerID="cri-o://27d71f7baa8f0db20c44110fc13599299b6aa2da2f4536bf654b2567c735abbc" gracePeriod=30 Oct 30 09:32:04.059041 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.059018 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:32:04.199037 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.199003 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z"] Oct 30 09:32:04.199599 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.199574 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0daeb10f-9b40-4ecb-bccd-d03c06be57c7" containerName="discovery" Oct 30 09:32:04.199655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.199606 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0daeb10f-9b40-4ecb-bccd-d03c06be57c7" containerName="discovery" Oct 30 09:32:04.199725 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.199714 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0daeb10f-9b40-4ecb-bccd-d03c06be57c7" containerName="discovery" Oct 30 09:32:04.203197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.203148 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" Oct 30 09:32:04.209870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.209848 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-kubeconfig\") pod \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " Oct 30 09:32:04.210001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.209891 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-kube-api-access-2dksv\" (UniqueName: \"kubernetes.io/projected/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-kube-api-access-2dksv\") pod \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " Oct 30 09:32:04.210001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.209917 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-csr-dns-cert\") pod \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " Oct 30 09:32:04.210001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.209941 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-cacerts\" (UniqueName: \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-cacerts\") pod \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " Oct 30 09:32:04.210201 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.210065 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-csr-ca-configmap\") pod \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " Oct 30 09:32:04.210201 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.210118 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-token\" (UniqueName: \"kubernetes.io/projected/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-token\") pod \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " Oct 30 09:32:04.210508 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.210216 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-local-certs\") pod \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\" (UID: \"0daeb10f-9b40-4ecb-bccd-d03c06be57c7\") " Oct 30 09:32:04.210698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.210661 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-csr-ca-configmap" pod "0daeb10f-9b40-4ecb-bccd-d03c06be57c7" (UID: "0daeb10f-9b40-4ecb-bccd-d03c06be57c7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:32:04.210959 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.210819 5938 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-csr-ca-configmap\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:04.215534 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.215500 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-csr-dns-cert" pod "0daeb10f-9b40-4ecb-bccd-d03c06be57c7" (UID: "0daeb10f-9b40-4ecb-bccd-d03c06be57c7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:32:04.217655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.217006 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-kubeconfig" pod "0daeb10f-9b40-4ecb-bccd-d03c06be57c7" (UID: "0daeb10f-9b40-4ecb-bccd-d03c06be57c7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:32:04.217655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.217071 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-cacerts" pod "0daeb10f-9b40-4ecb-bccd-d03c06be57c7" (UID: "0daeb10f-9b40-4ecb-bccd-d03c06be57c7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:32:04.217655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.217183 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-kube-api-access-2dksv" pod "0daeb10f-9b40-4ecb-bccd-d03c06be57c7" (UID: "0daeb10f-9b40-4ecb-bccd-d03c06be57c7"). InnerVolumeSpecName "kube-api-access-2dksv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:04.218022 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.217991 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-token" pod "0daeb10f-9b40-4ecb-bccd-d03c06be57c7" (UID: "0daeb10f-9b40-4ecb-bccd-d03c06be57c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:04.220977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.220954 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-local-certs" pod "0daeb10f-9b40-4ecb-bccd-d03c06be57c7" (UID: "0daeb10f-9b40-4ecb-bccd-d03c06be57c7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:04.227961 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.227940 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z"] Oct 30 09:32:04.311642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.311537 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f598e64d-7ecf-428d-a1dd-627df64307c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z\" (UID: \"f598e64d-7ecf-428d-a1dd-627df64307c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" Oct 30 09:32:04.311642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.311574 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f598e64d-7ecf-428d-a1dd-627df64307c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z\" (UID: \"f598e64d-7ecf-428d-a1dd-627df64307c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" Oct 30 09:32:04.311840 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.311798 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zmpb\" (UniqueName: \"kubernetes.io/projected/f598e64d-7ecf-428d-a1dd-627df64307c5-kube-api-access-6zmpb\") pod \"maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z\" (UID: \"f598e64d-7ecf-428d-a1dd-627df64307c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" Oct 30 09:32:04.311933 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.311919 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dksv\" (UniqueName: \"kubernetes.io/projected/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-kube-api-access-2dksv\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:04.311973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.311935 5938 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-csr-dns-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:04.311973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.311945 5938 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-cacerts\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:04.311973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.311956 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:04.311973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.311971 5938 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-local-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:04.312105 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.311980 5938 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0daeb10f-9b40-4ecb-bccd-d03c06be57c7-istio-kubeconfig\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:04.413721 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.413691 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmpb\" (UniqueName: \"kubernetes.io/projected/f598e64d-7ecf-428d-a1dd-627df64307c5-kube-api-access-6zmpb\") pod \"maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z\" (UID: \"f598e64d-7ecf-428d-a1dd-627df64307c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" Oct 30 09:32:04.413943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.413873 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f598e64d-7ecf-428d-a1dd-627df64307c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z\" (UID: \"f598e64d-7ecf-428d-a1dd-627df64307c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" Oct 30 09:32:04.413943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.413901 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f598e64d-7ecf-428d-a1dd-627df64307c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z\" (UID: \"f598e64d-7ecf-428d-a1dd-627df64307c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" Oct 30 09:32:04.414601 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.414567 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f598e64d-7ecf-428d-a1dd-627df64307c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z\" (UID: \"f598e64d-7ecf-428d-a1dd-627df64307c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" Oct 30 09:32:04.416648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.416619 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f598e64d-7ecf-428d-a1dd-627df64307c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z\" (UID: \"f598e64d-7ecf-428d-a1dd-627df64307c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" Oct 30 09:32:04.435774 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.435751 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmpb\" (UniqueName: \"kubernetes.io/projected/f598e64d-7ecf-428d-a1dd-627df64307c5-kube-api-access-6zmpb\") pod \"maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z\" (UID: \"f598e64d-7ecf-428d-a1dd-627df64307c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" Oct 30 09:32:04.522565 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.522529 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" Oct 30 09:32:04.705568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.705530 5938 generic.go:356] "Generic (PLEG): container finished" podID="0daeb10f-9b40-4ecb-bccd-d03c06be57c7" containerID="27d71f7baa8f0db20c44110fc13599299b6aa2da2f4536bf654b2567c735abbc" exitCode=0 Oct 30 09:32:04.706044 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.705580 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" Oct 30 09:32:04.706044 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.705624 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" event={"ID":"0daeb10f-9b40-4ecb-bccd-d03c06be57c7","Type":"ContainerDied","Data":"27d71f7baa8f0db20c44110fc13599299b6aa2da2f4536bf654b2567c735abbc"} Oct 30 09:32:04.706044 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.705679 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g" event={"ID":"0daeb10f-9b40-4ecb-bccd-d03c06be57c7","Type":"ContainerDied","Data":"270fed02e4f4b8fffef655e4077e21e8ea06045db780646b51e132e4cebf404f"} Oct 30 09:32:04.706044 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.705701 5938 scope.go:122] "RemoveContainer" containerID="27d71f7baa8f0db20c44110fc13599299b6aa2da2f4536bf654b2567c735abbc" Oct 30 09:32:04.716814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.716574 5938 scope.go:122] "RemoveContainer" containerID="27d71f7baa8f0db20c44110fc13599299b6aa2da2f4536bf654b2567c735abbc" Oct 30 09:32:04.716938 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:32:04.716917 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d71f7baa8f0db20c44110fc13599299b6aa2da2f4536bf654b2567c735abbc\": container with ID starting with 27d71f7baa8f0db20c44110fc13599299b6aa2da2f4536bf654b2567c735abbc not found: ID does not exist" containerID="27d71f7baa8f0db20c44110fc13599299b6aa2da2f4536bf654b2567c735abbc" Oct 30 09:32:04.717004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.716946 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d71f7baa8f0db20c44110fc13599299b6aa2da2f4536bf654b2567c735abbc"} err="failed to get container status \"27d71f7baa8f0db20c44110fc13599299b6aa2da2f4536bf654b2567c735abbc\": rpc error: code = NotFound desc = could not find container \"27d71f7baa8f0db20c44110fc13599299b6aa2da2f4536bf654b2567c735abbc\": container with ID starting with 27d71f7baa8f0db20c44110fc13599299b6aa2da2f4536bf654b2567c735abbc not found: ID does not exist" Oct 30 09:32:04.764946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.764922 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g"] Oct 30 09:32:04.770109 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.770079 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fw98g"] Oct 30 09:32:04.829777 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:32:04.829707 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf598e64d_7ecf_428d_a1dd_627df64307c5.slice/crio-24ba1a933b8be1ad64cd668de7861eb8ca8337880dbbbafa28768545acea3f46 WatchSource:0}: Error finding container 24ba1a933b8be1ad64cd668de7861eb8ca8337880dbbbafa28768545acea3f46: Status 404 returned error can't find the container with id 24ba1a933b8be1ad64cd668de7861eb8ca8337880dbbbafa28768545acea3f46 Oct 30 09:32:04.835304 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.835268 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z"] Oct 30 09:32:04.856446 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:04.856423 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf" Oct 30 09:32:05.020079 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.020009 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-kube-api-access-kwxd2\" (UniqueName: \"kubernetes.io/projected/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-kube-api-access-kwxd2\") pod \"4f98c0a5-33ff-4e3b-946d-1b50bc80502b\" (UID: \"4f98c0a5-33ff-4e3b-946d-1b50bc80502b\") " Oct 30 09:32:05.020079 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.020060 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-maas-api-openshift-ca\") pod \"4f98c0a5-33ff-4e3b-946d-1b50bc80502b\" (UID: \"4f98c0a5-33ff-4e3b-946d-1b50bc80502b\") " Oct 30 09:32:05.020337 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.020219 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-maas-api-xks-ca\") pod \"4f98c0a5-33ff-4e3b-946d-1b50bc80502b\" (UID: \"4f98c0a5-33ff-4e3b-946d-1b50bc80502b\") " Oct 30 09:32:05.020768 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.020708 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-maas-api-openshift-ca" pod "4f98c0a5-33ff-4e3b-946d-1b50bc80502b" (UID: "4f98c0a5-33ff-4e3b-946d-1b50bc80502b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:32:05.023285 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.023256 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-kube-api-access-kwxd2" pod "4f98c0a5-33ff-4e3b-946d-1b50bc80502b" (UID: "4f98c0a5-33ff-4e3b-946d-1b50bc80502b"). InnerVolumeSpecName "kube-api-access-kwxd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:05.029795 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.029766 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-maas-api-xks-ca" pod "4f98c0a5-33ff-4e3b-946d-1b50bc80502b" (UID: "4f98c0a5-33ff-4e3b-946d-1b50bc80502b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:32:05.124789 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.124754 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:05.125515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.125484 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwxd2\" (UniqueName: \"kubernetes.io/projected/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-kube-api-access-kwxd2\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:05.125701 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.125677 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f98c0a5-33ff-4e3b-946d-1b50bc80502b-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:05.719335 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.719293 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" event={"ID":"f598e64d-7ecf-428d-a1dd-627df64307c5","Type":"ContainerStarted","Data":"d7d5b1a093fcfa6c669c88137077a83d1bbe5ab29156091e0f28a457921cc1e4"} Oct 30 09:32:05.721040 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.719986 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" event={"ID":"f598e64d-7ecf-428d-a1dd-627df64307c5","Type":"ContainerStarted","Data":"24ba1a933b8be1ad64cd668de7861eb8ca8337880dbbbafa28768545acea3f46"} Oct 30 09:32:05.723140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.723114 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf" event={"ID":"4f98c0a5-33ff-4e3b-946d-1b50bc80502b","Type":"ContainerDied","Data":"b1c0446a9b871b30a48f76c2f3e2a0be1560877615437fd7a21cd75ae09d30ab"} Oct 30 09:32:05.723305 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.723289 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1c0446a9b871b30a48f76c2f3e2a0be1560877615437fd7a21cd75ae09d30ab" Oct 30 09:32:05.723474 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.723461 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf" Oct 30 09:32:05.745926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.745885 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27"] Oct 30 09:32:05.747093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.747066 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4f98c0a5-33ff-4e3b-946d-1b50bc80502b" containerName="revoke-keys" Oct 30 09:32:05.747270 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.747255 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f98c0a5-33ff-4e3b-946d-1b50bc80502b" containerName="revoke-keys" Oct 30 09:32:05.747578 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.747560 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f98c0a5-33ff-4e3b-946d-1b50bc80502b" containerName="revoke-keys" Oct 30 09:32:05.840345 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.840251 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" podStartSLOduration=1.840222107 podStartE2EDuration="1.840222107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:05.818898853 +0000 UTC m=+2243.681604674" watchObservedRunningTime="2026-10-30 09:32:05.840222107 +0000 UTC m=+2243.702927932" Oct 30 09:32:05.906740 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.883060 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27"] Oct 30 09:32:05.906740 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.883240 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" Oct 30 09:32:05.947786 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.947722 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pffj\" (UniqueName: \"kubernetes.io/projected/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-kube-api-access-6pffj\") pod \"maas-api-revoke-sub-ecbf83a1-fpc27\" (UID: \"80bb7eaf-6ff2-4138-8a40-c5ed799a1123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" Oct 30 09:32:05.948293 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.948268 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecbf83a1-fpc27\" (UID: \"80bb7eaf-6ff2-4138-8a40-c5ed799a1123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" Oct 30 09:32:05.948674 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:05.948638 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecbf83a1-fpc27\" (UID: \"80bb7eaf-6ff2-4138-8a40-c5ed799a1123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" Oct 30 09:32:06.050050 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:06.049945 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecbf83a1-fpc27\" (UID: \"80bb7eaf-6ff2-4138-8a40-c5ed799a1123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" Oct 30 09:32:06.050991 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:06.050961 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecbf83a1-fpc27\" (UID: \"80bb7eaf-6ff2-4138-8a40-c5ed799a1123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" Oct 30 09:32:06.051217 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:06.051198 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pffj\" (UniqueName: \"kubernetes.io/projected/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-kube-api-access-6pffj\") pod \"maas-api-revoke-sub-ecbf83a1-fpc27\" (UID: \"80bb7eaf-6ff2-4138-8a40-c5ed799a1123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" Oct 30 09:32:06.069611 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:06.054424 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecbf83a1-fpc27\" (UID: \"80bb7eaf-6ff2-4138-8a40-c5ed799a1123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" Oct 30 09:32:06.073611 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:06.072368 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecbf83a1-fpc27\" (UID: \"80bb7eaf-6ff2-4138-8a40-c5ed799a1123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" Oct 30 09:32:06.132619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:06.124262 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pffj\" (UniqueName: \"kubernetes.io/projected/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-kube-api-access-6pffj\") pod \"maas-api-revoke-sub-ecbf83a1-fpc27\" (UID: \"80bb7eaf-6ff2-4138-8a40-c5ed799a1123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" Oct 30 09:32:06.262941 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:06.262909 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" Oct 30 09:32:06.287186 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:06.287152 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0daeb10f-9b40-4ecb-bccd-d03c06be57c7" path="/var/lib/kubelet/pods/0daeb10f-9b40-4ecb-bccd-d03c06be57c7/volumes" Oct 30 09:32:07.073795 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:07.073688 5938 generic.go:356] "Generic (PLEG): container finished" podID="f598e64d-7ecf-428d-a1dd-627df64307c5" containerID="d7d5b1a093fcfa6c669c88137077a83d1bbe5ab29156091e0f28a457921cc1e4" exitCode=0 Oct 30 09:32:07.074680 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:07.074640 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" event={"ID":"f598e64d-7ecf-428d-a1dd-627df64307c5","Type":"ContainerDied","Data":"d7d5b1a093fcfa6c669c88137077a83d1bbe5ab29156091e0f28a457921cc1e4"} Oct 30 09:32:07.191027 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:07.184716 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27"] Oct 30 09:32:08.084016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:08.083924 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" event={"ID":"80bb7eaf-6ff2-4138-8a40-c5ed799a1123","Type":"ContainerStarted","Data":"ce50d9d5a6ab7e0c739e1b42e6649fadc9bc738f4fccd56fe39649aad5674cc3"} Oct 30 09:32:08.084016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:08.083988 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" event={"ID":"80bb7eaf-6ff2-4138-8a40-c5ed799a1123","Type":"ContainerStarted","Data":"993392f312c9095b586095506af57e40c6f409e965c17d98cfb6315f008934db"} Oct 30 09:32:08.128099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:08.128046 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" podStartSLOduration=3.128030838 podStartE2EDuration="3.128030838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:08.125749903 +0000 UTC m=+2245.988455725" watchObservedRunningTime="2026-10-30 09:32:08.128030838 +0000 UTC m=+2245.990736659" Oct 30 09:32:08.244878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:08.244852 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" Oct 30 09:32:08.310359 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:08.310324 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f598e64d-7ecf-428d-a1dd-627df64307c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f598e64d-7ecf-428d-a1dd-627df64307c5-maas-api-xks-ca\") pod \"f598e64d-7ecf-428d-a1dd-627df64307c5\" (UID: \"f598e64d-7ecf-428d-a1dd-627df64307c5\") " Oct 30 09:32:08.310530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:08.310413 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f598e64d-7ecf-428d-a1dd-627df64307c5-kube-api-access-6zmpb\" (UniqueName: \"kubernetes.io/projected/f598e64d-7ecf-428d-a1dd-627df64307c5-kube-api-access-6zmpb\") pod \"f598e64d-7ecf-428d-a1dd-627df64307c5\" (UID: \"f598e64d-7ecf-428d-a1dd-627df64307c5\") " Oct 30 09:32:08.310530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:08.310472 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f598e64d-7ecf-428d-a1dd-627df64307c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f598e64d-7ecf-428d-a1dd-627df64307c5-maas-api-openshift-ca\") pod \"f598e64d-7ecf-428d-a1dd-627df64307c5\" (UID: \"f598e64d-7ecf-428d-a1dd-627df64307c5\") " Oct 30 09:32:08.310872 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:08.310841 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f598e64d-7ecf-428d-a1dd-627df64307c5-maas-api-openshift-ca" pod "f598e64d-7ecf-428d-a1dd-627df64307c5" (UID: "f598e64d-7ecf-428d-a1dd-627df64307c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:32:08.311122 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:08.311104 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f598e64d-7ecf-428d-a1dd-627df64307c5-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:08.312664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:08.312641 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f598e64d-7ecf-428d-a1dd-627df64307c5-kube-api-access-6zmpb" pod "f598e64d-7ecf-428d-a1dd-627df64307c5" (UID: "f598e64d-7ecf-428d-a1dd-627df64307c5"). InnerVolumeSpecName "kube-api-access-6zmpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:08.312788 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:08.312771 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f598e64d-7ecf-428d-a1dd-627df64307c5-maas-api-xks-ca" pod "f598e64d-7ecf-428d-a1dd-627df64307c5" (UID: "f598e64d-7ecf-428d-a1dd-627df64307c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:32:08.412111 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:08.412081 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f598e64d-7ecf-428d-a1dd-627df64307c5-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:08.412111 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:08.412107 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zmpb\" (UniqueName: \"kubernetes.io/projected/f598e64d-7ecf-428d-a1dd-627df64307c5-kube-api-access-6zmpb\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:09.094194 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:09.094161 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" Oct 30 09:32:09.094194 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:09.094169 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z" event={"ID":"f598e64d-7ecf-428d-a1dd-627df64307c5","Type":"ContainerDied","Data":"24ba1a933b8be1ad64cd668de7861eb8ca8337880dbbbafa28768545acea3f46"} Oct 30 09:32:09.094782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:09.094230 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24ba1a933b8be1ad64cd668de7861eb8ca8337880dbbbafa28768545acea3f46" Oct 30 09:32:09.095926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:09.095888 5938 generic.go:356] "Generic (PLEG): container finished" podID="80bb7eaf-6ff2-4138-8a40-c5ed799a1123" containerID="ce50d9d5a6ab7e0c739e1b42e6649fadc9bc738f4fccd56fe39649aad5674cc3" exitCode=0 Oct 30 09:32:09.096049 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:09.095921 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" event={"ID":"80bb7eaf-6ff2-4138-8a40-c5ed799a1123","Type":"ContainerDied","Data":"ce50d9d5a6ab7e0c739e1b42e6649fadc9bc738f4fccd56fe39649aad5674cc3"} Oct 30 09:32:10.264063 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:10.264032 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" Oct 30 09:32:10.337433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:10.337400 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-maas-api-xks-ca\") pod \"80bb7eaf-6ff2-4138-8a40-c5ed799a1123\" (UID: \"80bb7eaf-6ff2-4138-8a40-c5ed799a1123\") " Oct 30 09:32:10.337666 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:10.337507 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-kube-api-access-6pffj\" (UniqueName: \"kubernetes.io/projected/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-kube-api-access-6pffj\") pod \"80bb7eaf-6ff2-4138-8a40-c5ed799a1123\" (UID: \"80bb7eaf-6ff2-4138-8a40-c5ed799a1123\") " Oct 30 09:32:10.337666 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:10.337654 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-maas-api-openshift-ca\") pod \"80bb7eaf-6ff2-4138-8a40-c5ed799a1123\" (UID: \"80bb7eaf-6ff2-4138-8a40-c5ed799a1123\") " Oct 30 09:32:10.338006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:10.337978 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-maas-api-openshift-ca" pod "80bb7eaf-6ff2-4138-8a40-c5ed799a1123" (UID: "80bb7eaf-6ff2-4138-8a40-c5ed799a1123"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:32:10.338285 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:10.338265 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:10.340365 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:10.340341 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-maas-api-xks-ca" pod "80bb7eaf-6ff2-4138-8a40-c5ed799a1123" (UID: "80bb7eaf-6ff2-4138-8a40-c5ed799a1123"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:32:10.340495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:10.340364 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-kube-api-access-6pffj" pod "80bb7eaf-6ff2-4138-8a40-c5ed799a1123" (UID: "80bb7eaf-6ff2-4138-8a40-c5ed799a1123"). InnerVolumeSpecName "kube-api-access-6pffj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:10.439418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:10.439381 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:10.439418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:10.439412 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pffj\" (UniqueName: \"kubernetes.io/projected/80bb7eaf-6ff2-4138-8a40-c5ed799a1123-kube-api-access-6pffj\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:11.111906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.111878 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" Oct 30 09:32:11.112096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.111917 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27" event={"ID":"80bb7eaf-6ff2-4138-8a40-c5ed799a1123","Type":"ContainerDied","Data":"993392f312c9095b586095506af57e40c6f409e965c17d98cfb6315f008934db"} Oct 30 09:32:11.112096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.111964 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="993392f312c9095b586095506af57e40c6f409e965c17d98cfb6315f008934db" Oct 30 09:32:11.719183 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.719152 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28"] Oct 30 09:32:11.719703 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.719688 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f598e64d-7ecf-428d-a1dd-627df64307c5" containerName="revoke-keys" Oct 30 09:32:11.719703 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.719703 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f598e64d-7ecf-428d-a1dd-627df64307c5" containerName="revoke-keys" Oct 30 09:32:11.719824 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.719738 5938 cpu_manager.go:414] "containerMap: removing container" podUID="80bb7eaf-6ff2-4138-8a40-c5ed799a1123" containerName="revoke-keys" Oct 30 09:32:11.719824 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.719743 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80bb7eaf-6ff2-4138-8a40-c5ed799a1123" containerName="revoke-keys" Oct 30 09:32:11.719891 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.719859 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f598e64d-7ecf-428d-a1dd-627df64307c5" containerName="revoke-keys" Oct 30 09:32:11.719891 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.719874 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="80bb7eaf-6ff2-4138-8a40-c5ed799a1123" containerName="revoke-keys" Oct 30 09:32:11.723420 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.723401 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.743047 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.743023 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d87c85-openshift-default-dockercfg-cjd75\"" Oct 30 09:32:11.755779 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.755749 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-workload-certs\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.755945 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.755824 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-podinfo\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.755945 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.755922 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-credential-socket\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.756071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.755946 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istiod-ca-cert\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.756071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.755968 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-envoy\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.756071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.755988 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-workload-socket\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.756226 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.756124 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-token\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.756226 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.756172 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnqqw\" (UniqueName: \"kubernetes.io/projected/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-kube-api-access-pnqqw\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.756338 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.756224 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-data\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.762009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.761979 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28"] Oct 30 09:32:11.857177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.857145 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnqqw\" (UniqueName: \"kubernetes.io/projected/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-kube-api-access-pnqqw\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.857343 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.857211 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-data\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.857343 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.857251 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-workload-certs\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.857451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.857394 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-podinfo\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.857505 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.857487 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-credential-socket\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.857557 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.857541 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istiod-ca-cert\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.857624 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.857568 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-envoy\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.857690 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.857629 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-workload-socket\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.857690 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.857647 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-data\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.857799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.857711 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-workload-certs\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.857799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.857785 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-token\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.857919 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.857794 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-credential-socket\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.858016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.857990 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-workload-socket\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.858322 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.858271 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istiod-ca-cert\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.859863 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.859841 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-envoy\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.860109 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.860090 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-podinfo\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.889279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.889253 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-token\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:11.889422 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:11.889405 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnqqw\" (UniqueName: \"kubernetes.io/projected/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-kube-api-access-pnqqw\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:12.037292 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:12.037209 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:12.360386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:12.360353 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28"] Oct 30 09:32:12.361210 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:32:12.361181 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22f8b1f_a12d_4031_98eb_8acb7b00c1fb.slice/crio-0facc7ab00a999067a4150863819566304b1f64d94db789876920a73c767e613 WatchSource:0}: Error finding container 0facc7ab00a999067a4150863819566304b1f64d94db789876920a73c767e613: Status 404 returned error can't find the container with id 0facc7ab00a999067a4150863819566304b1f64d94db789876920a73c767e613 Oct 30 09:32:12.363991 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:12.363958 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:32:12.364077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:12.364025 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:32:12.364077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:12.364057 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:32:13.135981 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.135944 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" event={"ID":"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb","Type":"ContainerStarted","Data":"6e8cecce79859d2544d3cefdf645c96d98b92200c52b9df01403c45033c73214"} Oct 30 09:32:13.136623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.136568 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" event={"ID":"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb","Type":"ContainerStarted","Data":"0facc7ab00a999067a4150863819566304b1f64d94db789876920a73c767e613"} Oct 30 09:32:13.180788 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.180760 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn"] Oct 30 09:32:13.184878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.184855 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.205854 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.205743 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-fc7ee7-openshift-default-dockercfg-xtmjd\"" Oct 30 09:32:13.208383 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.208356 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn"] Oct 30 09:32:13.217437 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.217375 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" podStartSLOduration=2.217356767 podStartE2EDuration="2.217356767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:13.216383002 +0000 UTC m=+2251.079088836" watchObservedRunningTime="2026-10-30 09:32:13.217356767 +0000 UTC m=+2251.080062589" Oct 30 09:32:13.276444 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.276407 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-envoy\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.276695 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.276494 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istiod-ca-cert\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.276695 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.276546 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-credential-socket\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.276695 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.276632 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-workload-certs\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.276695 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.276665 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d79hh\" (UniqueName: \"kubernetes.io/projected/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-kube-api-access-d79hh\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.276940 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.276768 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-workload-socket\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.276940 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.276815 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-podinfo\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.276940 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.276915 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-token\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.277097 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.276971 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-data\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.381612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.380753 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-token\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.381612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.380833 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-data\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.381612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.380879 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-envoy\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.381612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.380947 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istiod-ca-cert\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.381612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.380994 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-credential-socket\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.381612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.381061 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-workload-certs\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.381612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.381090 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d79hh\" (UniqueName: \"kubernetes.io/projected/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-kube-api-access-d79hh\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.381612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.381200 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-workload-socket\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.381612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.381250 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-podinfo\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.393068 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.384007 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-workload-certs\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.393068 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.384293 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-credential-socket\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.393068 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.384542 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-data\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.394290 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.394258 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-workload-socket\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.404578 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.404540 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istiod-ca-cert\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.435139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.435106 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-podinfo\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.456777 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.456738 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-token\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.457665 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.457636 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d79hh\" (UniqueName: \"kubernetes.io/projected/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-kube-api-access-d79hh\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.458320 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.458297 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-envoy\") pod \"e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.507004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.505871 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:13.907957 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.907922 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn"] Oct 30 09:32:13.909356 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:32:13.909325 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2573c7_dd0a_4ba6_8e74_66cd5eef4361.slice/crio-0a1f8b5cbb669e68237801ee85414a4a98d7dff6113325f1077750c5bd63ec5d WatchSource:0}: Error finding container 0a1f8b5cbb669e68237801ee85414a4a98d7dff6113325f1077750c5bd63ec5d: Status 404 returned error can't find the container with id 0a1f8b5cbb669e68237801ee85414a4a98d7dff6113325f1077750c5bd63ec5d Oct 30 09:32:13.912231 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.912183 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:32:13.912329 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.912281 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:32:13.912516 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:13.912328 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:32:14.055610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:14.042146 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:14.055610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:14.042319 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 09:32:14.055610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:14.042380 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" podUID="c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 09:32:14.161915 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:14.161739 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" event={"ID":"be2573c7-dd0a-4ba6-8e74-66cd5eef4361","Type":"ContainerStarted","Data":"3d9a61ef06c9603c8464098f20d12f669ba8b8640bebbf1de71840e54a7aabb6"} Oct 30 09:32:14.161915 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:14.161814 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" event={"ID":"be2573c7-dd0a-4ba6-8e74-66cd5eef4361","Type":"ContainerStarted","Data":"0a1f8b5cbb669e68237801ee85414a4a98d7dff6113325f1077750c5bd63ec5d"} Oct 30 09:32:14.210393 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:14.210321 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" podStartSLOduration=1.210303777 podStartE2EDuration="1.210303777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:14.206385131 +0000 UTC m=+2252.069090952" watchObservedRunningTime="2026-10-30 09:32:14.210303777 +0000 UTC m=+2252.073009600" Oct 30 09:32:14.509279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:14.509183 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:14.683618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:14.683567 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t"] Oct 30 09:32:14.718434 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:14.718402 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql"] Oct 30 09:32:15.040617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:15.040062 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 09:32:15.040617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:15.040123 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" podUID="c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 09:32:15.516756 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:15.516720 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:15.852331 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:32:15.852090 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:32:16.040615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:16.040474 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 09:32:16.040615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:16.040540 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" podUID="c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 09:32:16.205757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:16.205718 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:16.210373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:16.210344 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:32:17.041111 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:17.041076 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 09:32:17.041798 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:17.041762 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" podUID="c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 09:32:18.037890 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.037855 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 09:32:18.038199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.038150 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" podUID="c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 09:32:18.541854 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.541807 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr"] Oct 30 09:32:18.553186 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.553152 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.571254 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.570264 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr"] Oct 30 09:32:18.580648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.575479 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-fb1a2a-openshift-default-dockercfg-cn24l\"" Oct 30 09:32:18.658301 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.658251 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-workload-certs\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.658491 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.658354 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.658581 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.658555 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-envoy\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.658750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.658727 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-workload-socket\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.658818 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.658765 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-podinfo\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.658994 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.658955 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6zrv\" (UniqueName: \"kubernetes.io/projected/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-kube-api-access-h6zrv\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.659101 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.659079 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-credential-socket\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.659147 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.659133 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-data\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.659201 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.659191 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-token\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.838667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.837713 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6zrv\" (UniqueName: \"kubernetes.io/projected/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-kube-api-access-h6zrv\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.838667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.837773 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-credential-socket\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.838667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.837794 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-data\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.838667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.837816 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-token\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.838667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.837850 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-workload-certs\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.838667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.837892 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.838667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.837953 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-envoy\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.838667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.838013 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-workload-socket\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.838667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.838027 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-podinfo\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.869687 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.862508 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-data\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.869687 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.863233 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-credential-socket\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.869687 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.864006 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.869687 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.864347 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-workload-socket\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.871018 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.870945 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-workload-certs\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.881617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.878827 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-envoy\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.907847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.905443 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-podinfo\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.910199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.910109 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-token\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:18.939465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:18.939432 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6zrv\" (UniqueName: \"kubernetes.io/projected/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-kube-api-access-h6zrv\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:19.037697 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:19.037659 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 09:32:19.037863 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:19.037723 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" podUID="c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 09:32:19.213890 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:19.213850 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:19.634385 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:19.634340 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr"] Oct 30 09:32:19.634980 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:32:19.634947 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e745f6_3ef0_4391_b1ec_2e0feb40ddfe.slice/crio-a49207c071e67adc34a19433ec5d1a9fc5fa183ecccbcc1cebd2333d73fdcc1c WatchSource:0}: Error finding container a49207c071e67adc34a19433ec5d1a9fc5fa183ecccbcc1cebd2333d73fdcc1c: Status 404 returned error can't find the container with id a49207c071e67adc34a19433ec5d1a9fc5fa183ecccbcc1cebd2333d73fdcc1c Oct 30 09:32:19.637871 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:19.637833 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:32:19.637979 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:19.637904 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:32:19.637979 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:19.637936 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:32:19.684804 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:19.684726 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" podUID="ecc771d3-8e16-4e67-ab32-8c25d4326e86" containerName="payload-processing" containerID="cri-o://a740f460989e6bac23e448d3e52153c081564229825b7ac9652e36aa7701f11c" gracePeriod=25 Oct 30 09:32:19.718807 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:19.718746 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" podUID="0f465c90-efbc-4ce7-9dba-bf62492777c9" containerName="payload-pre-processing" containerID="cri-o://798296cfe1871e2575395771675a30e27926211b984de6ce40e9eff4490dd8e5" gracePeriod=25 Oct 30 09:32:20.060459 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.060396 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:20.063003 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.062931 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:20.064832 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.064530 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:32:20.089722 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.089690 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" Oct 30 09:32:20.094272 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.094240 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" Oct 30 09:32:20.185699 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.185575 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc771d3-8e16-4e67-ab32-8c25d4326e86-kube-api-access-f562v\" (UniqueName: \"kubernetes.io/projected/ecc771d3-8e16-4e67-ab32-8c25d4326e86-kube-api-access-f562v\") pod \"ecc771d3-8e16-4e67-ab32-8c25d4326e86\" (UID: \"ecc771d3-8e16-4e67-ab32-8c25d4326e86\") " Oct 30 09:32:20.186006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.185984 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc771d3-8e16-4e67-ab32-8c25d4326e86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc771d3-8e16-4e67-ab32-8c25d4326e86-plugins-config-volume\") pod \"ecc771d3-8e16-4e67-ab32-8c25d4326e86\" (UID: \"ecc771d3-8e16-4e67-ab32-8c25d4326e86\") " Oct 30 09:32:20.186398 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.186366 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc771d3-8e16-4e67-ab32-8c25d4326e86-plugins-config-volume" pod "ecc771d3-8e16-4e67-ab32-8c25d4326e86" (UID: "ecc771d3-8e16-4e67-ab32-8c25d4326e86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:32:20.187093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.187066 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f465c90-efbc-4ce7-9dba-bf62492777c9-plugins-config-volume" pod "0f465c90-efbc-4ce7-9dba-bf62492777c9" (UID: "0f465c90-efbc-4ce7-9dba-bf62492777c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:32:20.187202 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.187109 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f465c90-efbc-4ce7-9dba-bf62492777c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f465c90-efbc-4ce7-9dba-bf62492777c9-plugins-config-volume\") pod \"0f465c90-efbc-4ce7-9dba-bf62492777c9\" (UID: \"0f465c90-efbc-4ce7-9dba-bf62492777c9\") " Oct 30 09:32:20.187318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.187297 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f465c90-efbc-4ce7-9dba-bf62492777c9-kube-api-access-ggms6\" (UniqueName: \"kubernetes.io/projected/0f465c90-efbc-4ce7-9dba-bf62492777c9-kube-api-access-ggms6\") pod \"0f465c90-efbc-4ce7-9dba-bf62492777c9\" (UID: \"0f465c90-efbc-4ce7-9dba-bf62492777c9\") " Oct 30 09:32:20.188348 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.188319 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f465c90-efbc-4ce7-9dba-bf62492777c9-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:20.188438 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.188353 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc771d3-8e16-4e67-ab32-8c25d4326e86-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:20.188734 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.188707 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc771d3-8e16-4e67-ab32-8c25d4326e86-kube-api-access-f562v" pod "ecc771d3-8e16-4e67-ab32-8c25d4326e86" (UID: "ecc771d3-8e16-4e67-ab32-8c25d4326e86"). InnerVolumeSpecName "kube-api-access-f562v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:20.191499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.191466 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f465c90-efbc-4ce7-9dba-bf62492777c9-kube-api-access-ggms6" pod "0f465c90-efbc-4ce7-9dba-bf62492777c9" (UID: "0f465c90-efbc-4ce7-9dba-bf62492777c9"). InnerVolumeSpecName "kube-api-access-ggms6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:20.280753 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.280717 5938 generic.go:356] "Generic (PLEG): container finished" podID="0f465c90-efbc-4ce7-9dba-bf62492777c9" containerID="798296cfe1871e2575395771675a30e27926211b984de6ce40e9eff4490dd8e5" exitCode=0 Oct 30 09:32:20.281172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.281153 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" Oct 30 09:32:20.282804 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.282754 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" event={"ID":"0f465c90-efbc-4ce7-9dba-bf62492777c9","Type":"ContainerDied","Data":"798296cfe1871e2575395771675a30e27926211b984de6ce40e9eff4490dd8e5"} Oct 30 09:32:20.282930 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.282862 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql" event={"ID":"0f465c90-efbc-4ce7-9dba-bf62492777c9","Type":"ContainerDied","Data":"1758fd440467c644be4c77b97e433b778206e249e32282a2dd9e31519d316be7"} Oct 30 09:32:20.282930 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.282897 5938 scope.go:122] "RemoveContainer" containerID="798296cfe1871e2575395771675a30e27926211b984de6ce40e9eff4490dd8e5" Oct 30 09:32:20.284357 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.284325 5938 generic.go:356] "Generic (PLEG): container finished" podID="ecc771d3-8e16-4e67-ab32-8c25d4326e86" containerID="a740f460989e6bac23e448d3e52153c081564229825b7ac9652e36aa7701f11c" exitCode=0 Oct 30 09:32:20.317904 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.286302 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" event={"ID":"ecc771d3-8e16-4e67-ab32-8c25d4326e86","Type":"ContainerDied","Data":"a740f460989e6bac23e448d3e52153c081564229825b7ac9652e36aa7701f11c"} Oct 30 09:32:20.318157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.318135 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" event={"ID":"ecc771d3-8e16-4e67-ab32-8c25d4326e86","Type":"ContainerDied","Data":"386257aa93d640d281926c3c14d8b555095faf805f0f32a344abfa16a0511dcd"} Oct 30 09:32:20.318881 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.305680 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t" Oct 30 09:32:20.319486 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.290849 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggms6\" (UniqueName: \"kubernetes.io/projected/0f465c90-efbc-4ce7-9dba-bf62492777c9-kube-api-access-ggms6\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:20.319937 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.319923 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f562v\" (UniqueName: \"kubernetes.io/projected/ecc771d3-8e16-4e67-ab32-8c25d4326e86-kube-api-access-f562v\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:20.323894 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.323871 5938 scope.go:122] "RemoveContainer" containerID="798296cfe1871e2575395771675a30e27926211b984de6ce40e9eff4490dd8e5" Oct 30 09:32:20.324687 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:32:20.324659 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798296cfe1871e2575395771675a30e27926211b984de6ce40e9eff4490dd8e5\": container with ID starting with 798296cfe1871e2575395771675a30e27926211b984de6ce40e9eff4490dd8e5 not found: ID does not exist" containerID="798296cfe1871e2575395771675a30e27926211b984de6ce40e9eff4490dd8e5" Oct 30 09:32:20.324796 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.324701 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798296cfe1871e2575395771675a30e27926211b984de6ce40e9eff4490dd8e5"} err="failed to get container status \"798296cfe1871e2575395771675a30e27926211b984de6ce40e9eff4490dd8e5\": rpc error: code = NotFound desc = could not find container \"798296cfe1871e2575395771675a30e27926211b984de6ce40e9eff4490dd8e5\": container with ID starting with 798296cfe1871e2575395771675a30e27926211b984de6ce40e9eff4490dd8e5 not found: ID does not exist" Oct 30 09:32:20.324796 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.324725 5938 scope.go:122] "RemoveContainer" containerID="a740f460989e6bac23e448d3e52153c081564229825b7ac9652e36aa7701f11c" Oct 30 09:32:20.329624 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.329394 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" event={"ID":"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe","Type":"ContainerStarted","Data":"ec34ecee1d481d600b1cd060d276523bc3f8cf8e1526f700e85fa150bf40619d"} Oct 30 09:32:20.329624 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.329448 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" event={"ID":"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe","Type":"ContainerStarted","Data":"a49207c071e67adc34a19433ec5d1a9fc5fa183ecccbcc1cebd2333d73fdcc1c"} Oct 30 09:32:20.342203 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.342175 5938 scope.go:122] "RemoveContainer" containerID="a740f460989e6bac23e448d3e52153c081564229825b7ac9652e36aa7701f11c" Oct 30 09:32:20.342663 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:32:20.342634 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a740f460989e6bac23e448d3e52153c081564229825b7ac9652e36aa7701f11c\": container with ID starting with a740f460989e6bac23e448d3e52153c081564229825b7ac9652e36aa7701f11c not found: ID does not exist" containerID="a740f460989e6bac23e448d3e52153c081564229825b7ac9652e36aa7701f11c" Oct 30 09:32:20.342805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.342670 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a740f460989e6bac23e448d3e52153c081564229825b7ac9652e36aa7701f11c"} err="failed to get container status \"a740f460989e6bac23e448d3e52153c081564229825b7ac9652e36aa7701f11c\": rpc error: code = NotFound desc = could not find container \"a740f460989e6bac23e448d3e52153c081564229825b7ac9652e36aa7701f11c\": container with ID starting with a740f460989e6bac23e448d3e52153c081564229825b7ac9652e36aa7701f11c not found: ID does not exist" Oct 30 09:32:20.350160 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.350113 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql"] Oct 30 09:32:20.353972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.353942 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-066f95-68968b4fff-r99ql"] Oct 30 09:32:20.395075 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.395013 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" podStartSLOduration=2.394994002 podStartE2EDuration="2.394994002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:20.394723568 +0000 UTC m=+2258.257429389" watchObservedRunningTime="2026-10-30 09:32:20.394994002 +0000 UTC m=+2258.257699824" Oct 30 09:32:20.436183 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.436070 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t"] Oct 30 09:32:20.444073 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.444039 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-066f95-5476fdb77d-dbq5t"] Oct 30 09:32:20.991063 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:20.991017 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z"] Oct 30 09:32:21.007388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:21.007338 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-066f95-zlc5z"] Oct 30 09:32:21.215060 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:21.215028 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:21.221110 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:21.221082 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:21.339868 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:21.339765 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:21.340799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:21.340782 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:32:22.257436 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:22.257348 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f465c90-efbc-4ce7-9dba-bf62492777c9" path="/var/lib/kubelet/pods/0f465c90-efbc-4ce7-9dba-bf62492777c9/volumes" Oct 30 09:32:22.257861 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:22.257813 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc771d3-8e16-4e67-ab32-8c25d4326e86" path="/var/lib/kubelet/pods/ecc771d3-8e16-4e67-ab32-8c25d4326e86/volumes" Oct 30 09:32:22.258207 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:22.258192 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f598e64d-7ecf-428d-a1dd-627df64307c5" path="/var/lib/kubelet/pods/f598e64d-7ecf-428d-a1dd-627df64307c5/volumes" Oct 30 09:32:24.852685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:24.852648 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s"] Oct 30 09:32:24.853241 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:24.853224 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ecc771d3-8e16-4e67-ab32-8c25d4326e86" containerName="payload-processing" Oct 30 09:32:24.853283 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:24.853241 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc771d3-8e16-4e67-ab32-8c25d4326e86" containerName="payload-processing" Oct 30 09:32:24.853283 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:24.853257 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0f465c90-efbc-4ce7-9dba-bf62492777c9" containerName="payload-pre-processing" Oct 30 09:32:24.853283 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:24.853262 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f465c90-efbc-4ce7-9dba-bf62492777c9" containerName="payload-pre-processing" Oct 30 09:32:24.853382 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:24.853369 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f465c90-efbc-4ce7-9dba-bf62492777c9" containerName="payload-pre-processing" Oct 30 09:32:24.853412 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:24.853386 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc771d3-8e16-4e67-ab32-8c25d4326e86" containerName="payload-processing" Oct 30 09:32:24.858485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:24.858460 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" Oct 30 09:32:24.875697 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:24.875660 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s"] Oct 30 09:32:24.884684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:24.884638 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-fb1a2a-dockercfg-wb5gn\"" Oct 30 09:32:24.884684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:24.884655 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-fb1a2a\"" Oct 30 09:32:24.911763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:24.911722 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1404655b-b5d3-4fb9-b426-131de90077fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s\" (UID: \"1404655b-b5d3-4fb9-b426-131de90077fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" Oct 30 09:32:24.911973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:24.911886 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6kh9\" (UniqueName: \"kubernetes.io/projected/1404655b-b5d3-4fb9-b426-131de90077fc-kube-api-access-n6kh9\") pod \"payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s\" (UID: \"1404655b-b5d3-4fb9-b426-131de90077fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" Oct 30 09:32:24.991625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:24.991055 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh"] Oct 30 09:32:25.053309 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:25.053252 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh"] Oct 30 09:32:25.053516 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:25.053423 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" Oct 30 09:32:25.053955 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:25.053925 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1404655b-b5d3-4fb9-b426-131de90077fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s\" (UID: \"1404655b-b5d3-4fb9-b426-131de90077fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" Oct 30 09:32:25.054257 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:25.054210 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6kh9\" (UniqueName: \"kubernetes.io/projected/1404655b-b5d3-4fb9-b426-131de90077fc-kube-api-access-n6kh9\") pod \"payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s\" (UID: \"1404655b-b5d3-4fb9-b426-131de90077fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" Oct 30 09:32:25.055983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:25.055279 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1404655b-b5d3-4fb9-b426-131de90077fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s\" (UID: \"1404655b-b5d3-4fb9-b426-131de90077fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" Oct 30 09:32:25.095840 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:25.081726 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6kh9\" (UniqueName: \"kubernetes.io/projected/1404655b-b5d3-4fb9-b426-131de90077fc-kube-api-access-n6kh9\") pod \"payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s\" (UID: \"1404655b-b5d3-4fb9-b426-131de90077fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" Oct 30 09:32:25.155746 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:25.155705 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5zjn\" (UniqueName: \"kubernetes.io/projected/9e09ce8d-bfb1-4729-a557-d1d9685547c5-kube-api-access-d5zjn\") pod \"payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh\" (UID: \"9e09ce8d-bfb1-4729-a557-d1d9685547c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" Oct 30 09:32:25.155955 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:25.155761 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e09ce8d-bfb1-4729-a557-d1d9685547c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh\" (UID: \"9e09ce8d-bfb1-4729-a557-d1d9685547c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" Oct 30 09:32:25.174348 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:25.174312 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" Oct 30 09:32:25.265617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:25.262991 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5zjn\" (UniqueName: \"kubernetes.io/projected/9e09ce8d-bfb1-4729-a557-d1d9685547c5-kube-api-access-d5zjn\") pod \"payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh\" (UID: \"9e09ce8d-bfb1-4729-a557-d1d9685547c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" Oct 30 09:32:25.265617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:25.263113 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e09ce8d-bfb1-4729-a557-d1d9685547c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh\" (UID: \"9e09ce8d-bfb1-4729-a557-d1d9685547c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" Oct 30 09:32:25.265617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:25.264657 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e09ce8d-bfb1-4729-a557-d1d9685547c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh\" (UID: \"9e09ce8d-bfb1-4729-a557-d1d9685547c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" Oct 30 09:32:25.296637 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:25.291915 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5zjn\" (UniqueName: \"kubernetes.io/projected/9e09ce8d-bfb1-4729-a557-d1d9685547c5-kube-api-access-d5zjn\") pod \"payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh\" (UID: \"9e09ce8d-bfb1-4729-a557-d1d9685547c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" Oct 30 09:32:25.409376 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:25.409294 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" Oct 30 09:32:25.524786 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:25.520827 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s"] Oct 30 09:32:25.530032 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:32:25.529995 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1404655b_b5d3_4fb9_b426_131de90077fc.slice/crio-1127c5c85eca497587609044ffb55307570195f26ee580fb7bb0f333ee2cc1a9 WatchSource:0}: Error finding container 1127c5c85eca497587609044ffb55307570195f26ee580fb7bb0f333ee2cc1a9: Status 404 returned error can't find the container with id 1127c5c85eca497587609044ffb55307570195f26ee580fb7bb0f333ee2cc1a9 Oct 30 09:32:25.719693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:25.719655 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh"] Oct 30 09:32:25.720322 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:32:25.720292 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e09ce8d_bfb1_4729_a557_d1d9685547c5.slice/crio-6a30c42783b27a3ca4ca4a80116978ab9aa717b892605cd2543d852bdeb71c99 WatchSource:0}: Error finding container 6a30c42783b27a3ca4ca4a80116978ab9aa717b892605cd2543d852bdeb71c99: Status 404 returned error can't find the container with id 6a30c42783b27a3ca4ca4a80116978ab9aa717b892605cd2543d852bdeb71c99 Oct 30 09:32:26.386318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:26.386279 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" event={"ID":"1404655b-b5d3-4fb9-b426-131de90077fc","Type":"ContainerStarted","Data":"8e5e23047d5313c85fab6a6a96afe8b5737f1c3702977dc69b80b41a7055f30a"} Oct 30 09:32:26.386826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:26.386371 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" event={"ID":"1404655b-b5d3-4fb9-b426-131de90077fc","Type":"ContainerStarted","Data":"1127c5c85eca497587609044ffb55307570195f26ee580fb7bb0f333ee2cc1a9"} Oct 30 09:32:26.386826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:26.386502 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" Oct 30 09:32:26.388159 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:26.388131 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" event={"ID":"9e09ce8d-bfb1-4729-a557-d1d9685547c5","Type":"ContainerStarted","Data":"93bac2fd84e1433b25e64b5fbf98b618cb52c3439a85fa1641a641ac5ba8541f"} Oct 30 09:32:26.388300 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:26.388184 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" event={"ID":"9e09ce8d-bfb1-4729-a557-d1d9685547c5","Type":"ContainerStarted","Data":"6a30c42783b27a3ca4ca4a80116978ab9aa717b892605cd2543d852bdeb71c99"} Oct 30 09:32:26.388300 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:26.388228 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" Oct 30 09:32:26.429668 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:26.429603 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" podStartSLOduration=2.429570686 podStartE2EDuration="2.429570686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:26.427277282 +0000 UTC m=+2264.289983117" watchObservedRunningTime="2026-10-30 09:32:26.429570686 +0000 UTC m=+2264.292276506" Oct 30 09:32:26.470995 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:26.470940 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" podStartSLOduration=2.470923476 podStartE2EDuration="2.470923476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:26.468599455 +0000 UTC m=+2264.331305269" watchObservedRunningTime="2026-10-30 09:32:26.470923476 +0000 UTC m=+2264.333629297" Oct 30 09:32:37.399686 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:37.399652 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" Oct 30 09:32:37.400336 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:37.400134 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" Oct 30 09:32:38.403447 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:38.403394 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx"] Oct 30 09:32:38.403921 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:38.403688 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" podUID="d11017ce-5d73-4775-a489-463a118338dd" containerName="istio-proxy" containerID="cri-o://c5a985f77dbd5fc0f9fab9fb7f2992f6a9886c57f2598586e52ed030eacafe22" gracePeriod=30 Oct 30 09:32:41.459721 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.459672 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt"] Oct 30 09:32:41.464532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.464498 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" Oct 30 09:32:41.482117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.482085 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt"] Oct 30 09:32:41.491721 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.491685 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-7c88ae\"" Oct 30 09:32:41.505332 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.505299 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-7c88ae-dockercfg-gx8kz\"" Oct 30 09:32:41.551734 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.551698 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq"] Oct 30 09:32:41.564962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.564929 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" Oct 30 09:32:41.573043 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.573001 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba161e1-40bc-4ea2-858a-44b63d9bdc83-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt\" (UID: \"5ba161e1-40bc-4ea2-858a-44b63d9bdc83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" Oct 30 09:32:41.573554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.573529 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8gxp\" (UniqueName: \"kubernetes.io/projected/5ba161e1-40bc-4ea2-858a-44b63d9bdc83-kube-api-access-b8gxp\") pod \"payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt\" (UID: \"5ba161e1-40bc-4ea2-858a-44b63d9bdc83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" Oct 30 09:32:41.616019 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.615986 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq"] Oct 30 09:32:41.688805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.688326 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4172ae-9665-4707-97ab-6b3c82fb7242-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq\" (UID: \"2d4172ae-9665-4707-97ab-6b3c82fb7242\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" Oct 30 09:32:41.688805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.688392 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba161e1-40bc-4ea2-858a-44b63d9bdc83-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt\" (UID: \"5ba161e1-40bc-4ea2-858a-44b63d9bdc83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" Oct 30 09:32:41.688805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.688664 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h2gj\" (UniqueName: \"kubernetes.io/projected/2d4172ae-9665-4707-97ab-6b3c82fb7242-kube-api-access-5h2gj\") pod \"payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq\" (UID: \"2d4172ae-9665-4707-97ab-6b3c82fb7242\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" Oct 30 09:32:41.690749 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.689466 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gxp\" (UniqueName: \"kubernetes.io/projected/5ba161e1-40bc-4ea2-858a-44b63d9bdc83-kube-api-access-b8gxp\") pod \"payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt\" (UID: \"5ba161e1-40bc-4ea2-858a-44b63d9bdc83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" Oct 30 09:32:41.691733 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.691705 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba161e1-40bc-4ea2-858a-44b63d9bdc83-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt\" (UID: \"5ba161e1-40bc-4ea2-858a-44b63d9bdc83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" Oct 30 09:32:41.710732 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.710652 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8gxp\" (UniqueName: \"kubernetes.io/projected/5ba161e1-40bc-4ea2-858a-44b63d9bdc83-kube-api-access-b8gxp\") pod \"payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt\" (UID: \"5ba161e1-40bc-4ea2-858a-44b63d9bdc83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" Oct 30 09:32:41.780476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.780420 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" Oct 30 09:32:41.790555 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.790518 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4172ae-9665-4707-97ab-6b3c82fb7242-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq\" (UID: \"2d4172ae-9665-4707-97ab-6b3c82fb7242\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" Oct 30 09:32:41.790744 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.790651 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2gj\" (UniqueName: \"kubernetes.io/projected/2d4172ae-9665-4707-97ab-6b3c82fb7242-kube-api-access-5h2gj\") pod \"payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq\" (UID: \"2d4172ae-9665-4707-97ab-6b3c82fb7242\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" Oct 30 09:32:41.791345 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.791322 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4172ae-9665-4707-97ab-6b3c82fb7242-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq\" (UID: \"2d4172ae-9665-4707-97ab-6b3c82fb7242\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" Oct 30 09:32:41.813933 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.813893 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h2gj\" (UniqueName: \"kubernetes.io/projected/2d4172ae-9665-4707-97ab-6b3c82fb7242-kube-api-access-5h2gj\") pod \"payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq\" (UID: \"2d4172ae-9665-4707-97ab-6b3c82fb7242\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" Oct 30 09:32:41.918358 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:41.918322 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" Oct 30 09:32:42.090928 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:42.090887 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt"] Oct 30 09:32:42.288927 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:42.288823 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq"] Oct 30 09:32:42.654241 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:42.654186 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" event={"ID":"5ba161e1-40bc-4ea2-858a-44b63d9bdc83","Type":"ContainerStarted","Data":"123d2e119810a1b0cf78e17c3e550b596c80f2d406cca575a62d34206210910d"} Oct 30 09:32:42.654782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:42.654264 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" Oct 30 09:32:42.654782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:42.654287 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" event={"ID":"5ba161e1-40bc-4ea2-858a-44b63d9bdc83","Type":"ContainerStarted","Data":"65ee762ea0778b9f8bc5e69de542b0a12089dcdcc593e4f0953b06b99a245430"} Oct 30 09:32:42.656300 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:42.656260 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" event={"ID":"2d4172ae-9665-4707-97ab-6b3c82fb7242","Type":"ContainerStarted","Data":"d6f7eb481f337708f2d0f6d8eb6bb58d2de4de1d57b626e2aa48452d65ea9e51"} Oct 30 09:32:42.656430 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:42.656322 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" event={"ID":"2d4172ae-9665-4707-97ab-6b3c82fb7242","Type":"ContainerStarted","Data":"5a8770cd5122dff29c6cfeea061e99fd83f3b7e37ba464b9dc6ad3693c04bfe6"} Oct 30 09:32:42.656430 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:42.656347 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" Oct 30 09:32:42.697854 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:42.697802 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" podStartSLOduration=1.6977866640000001 podStartE2EDuration="1.697786664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:42.693774506 +0000 UTC m=+2280.556480329" watchObservedRunningTime="2026-10-30 09:32:42.697786664 +0000 UTC m=+2280.560492511" Oct 30 09:32:42.734933 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:42.734877 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" podStartSLOduration=1.7348573840000001 podStartE2EDuration="1.734857384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:42.734383379 +0000 UTC m=+2280.597089202" watchObservedRunningTime="2026-10-30 09:32:42.734857384 +0000 UTC m=+2280.597563205" Oct 30 09:32:43.665203 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.665159 5938 generic.go:356] "Generic (PLEG): container finished" podID="d11017ce-5d73-4775-a489-463a118338dd" containerID="c5a985f77dbd5fc0f9fab9fb7f2992f6a9886c57f2598586e52ed030eacafe22" exitCode=0 Oct 30 09:32:43.665672 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.665235 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" event={"ID":"d11017ce-5d73-4775-a489-463a118338dd","Type":"ContainerDied","Data":"c5a985f77dbd5fc0f9fab9fb7f2992f6a9886c57f2598586e52ed030eacafe22"} Oct 30 09:32:43.689421 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.689343 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:32:43.826286 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.826245 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-istio-data\") pod \"d11017ce-5d73-4775-a489-463a118338dd\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " Oct 30 09:32:43.826489 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.826356 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-workload-certs\") pod \"d11017ce-5d73-4775-a489-463a118338dd\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " Oct 30 09:32:43.826489 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.826407 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d11017ce-5d73-4775-a489-463a118338dd-kube-api-access-lp499\" (UniqueName: \"kubernetes.io/projected/d11017ce-5d73-4775-a489-463a118338dd-kube-api-access-lp499\") pod \"d11017ce-5d73-4775-a489-463a118338dd\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " Oct 30 09:32:43.826629 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.826493 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d11017ce-5d73-4775-a489-463a118338dd-istio-token\" (UniqueName: \"kubernetes.io/projected/d11017ce-5d73-4775-a489-463a118338dd-istio-token\") pod \"d11017ce-5d73-4775-a489-463a118338dd\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " Oct 30 09:32:43.826629 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.826516 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-credential-socket\") pod \"d11017ce-5d73-4775-a489-463a118338dd\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " Oct 30 09:32:43.826629 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.826534 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d11017ce-5d73-4775-a489-463a118338dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d11017ce-5d73-4775-a489-463a118338dd-istiod-ca-cert\") pod \"d11017ce-5d73-4775-a489-463a118338dd\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " Oct 30 09:32:43.826629 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.826607 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-istio-data" pod "d11017ce-5d73-4775-a489-463a118338dd" (UID: "d11017ce-5d73-4775-a489-463a118338dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:43.826833 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.826630 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-workload-certs" pod "d11017ce-5d73-4775-a489-463a118338dd" (UID: "d11017ce-5d73-4775-a489-463a118338dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:43.826833 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.826671 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-istio-envoy\") pod \"d11017ce-5d73-4775-a489-463a118338dd\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " Oct 30 09:32:43.826833 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.826714 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-workload-socket\") pod \"d11017ce-5d73-4775-a489-463a118338dd\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " Oct 30 09:32:43.826833 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.826751 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d11017ce-5d73-4775-a489-463a118338dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d11017ce-5d73-4775-a489-463a118338dd-istio-podinfo\") pod \"d11017ce-5d73-4775-a489-463a118338dd\" (UID: \"d11017ce-5d73-4775-a489-463a118338dd\") " Oct 30 09:32:43.827033 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.826892 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-credential-socket" pod "d11017ce-5d73-4775-a489-463a118338dd" (UID: "d11017ce-5d73-4775-a489-463a118338dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:43.827033 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.826936 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d11017ce-5d73-4775-a489-463a118338dd-istiod-ca-cert" pod "d11017ce-5d73-4775-a489-463a118338dd" (UID: "d11017ce-5d73-4775-a489-463a118338dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:32:43.827143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.827057 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-workload-socket" pod "d11017ce-5d73-4775-a489-463a118338dd" (UID: "d11017ce-5d73-4775-a489-463a118338dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:43.827486 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.827458 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:43.827667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.827491 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:43.827667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.827509 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:43.827800 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.827710 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d11017ce-5d73-4775-a489-463a118338dd-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:43.827800 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.827758 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:43.829448 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.829416 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d11017ce-5d73-4775-a489-463a118338dd-istio-podinfo" pod "d11017ce-5d73-4775-a489-463a118338dd" (UID: "d11017ce-5d73-4775-a489-463a118338dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:32:43.829448 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.829432 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11017ce-5d73-4775-a489-463a118338dd-istio-token" pod "d11017ce-5d73-4775-a489-463a118338dd" (UID: "d11017ce-5d73-4775-a489-463a118338dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:43.829718 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.829677 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-istio-envoy" pod "d11017ce-5d73-4775-a489-463a118338dd" (UID: "d11017ce-5d73-4775-a489-463a118338dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:43.830234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.830205 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11017ce-5d73-4775-a489-463a118338dd-kube-api-access-lp499" pod "d11017ce-5d73-4775-a489-463a118338dd" (UID: "d11017ce-5d73-4775-a489-463a118338dd"). InnerVolumeSpecName "kube-api-access-lp499". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:43.929534 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.929492 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp499\" (UniqueName: \"kubernetes.io/projected/d11017ce-5d73-4775-a489-463a118338dd-kube-api-access-lp499\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:43.929534 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.929529 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d11017ce-5d73-4775-a489-463a118338dd-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:43.929534 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.929539 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d11017ce-5d73-4775-a489-463a118338dd-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:43.929826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:43.929548 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d11017ce-5d73-4775-a489-463a118338dd-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:32:44.675735 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:44.675703 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" Oct 30 09:32:44.676271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:44.675740 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx" event={"ID":"d11017ce-5d73-4775-a489-463a118338dd","Type":"ContainerDied","Data":"f82f9034a86f3b520e45261a14832954e8145a60995b2a7e979e83509d86ca98"} Oct 30 09:32:44.676271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:44.675830 5938 scope.go:122] "RemoveContainer" containerID="c5a985f77dbd5fc0f9fab9fb7f2992f6a9886c57f2598586e52ed030eacafe22" Oct 30 09:32:44.757676 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:44.757638 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx"] Oct 30 09:32:44.761054 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:44.761028 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t2zhx"] Oct 30 09:32:45.811311 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:32:45.811275 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:32:46.258206 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:46.258161 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d11017ce-5d73-4775-a489-463a118338dd" path="/var/lib/kubelet/pods/d11017ce-5d73-4775-a489-463a118338dd/volumes" Oct 30 09:32:52.663575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:52.663534 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww"] Oct 30 09:32:52.686064 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:52.686005 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d11017ce-5d73-4775-a489-463a118338dd" containerName="istio-proxy" Oct 30 09:32:52.696599 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:52.696526 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d11017ce-5d73-4775-a489-463a118338dd" containerName="istio-proxy" Oct 30 09:32:52.697551 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:52.697491 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d11017ce-5d73-4775-a489-463a118338dd" containerName="istio-proxy" Oct 30 09:32:52.708784 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:52.708753 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww"] Oct 30 09:32:52.709337 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:52.709105 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" Oct 30 09:32:52.747458 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:52.747432 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-304459\"" Oct 30 09:32:52.747695 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:52.747433 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-304459-dockercfg-srp55\"" Oct 30 09:32:52.771352 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:52.771306 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqppz\" (UniqueName: \"kubernetes.io/projected/f8f88266-2ef9-4a30-bfee-89816bc75032-kube-api-access-nqppz\") pod \"payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww\" (UID: \"f8f88266-2ef9-4a30-bfee-89816bc75032\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" Oct 30 09:32:52.771547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:52.771468 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f88266-2ef9-4a30-bfee-89816bc75032-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww\" (UID: \"f8f88266-2ef9-4a30-bfee-89816bc75032\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" Oct 30 09:32:52.872462 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:52.872420 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqppz\" (UniqueName: \"kubernetes.io/projected/f8f88266-2ef9-4a30-bfee-89816bc75032-kube-api-access-nqppz\") pod \"payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww\" (UID: \"f8f88266-2ef9-4a30-bfee-89816bc75032\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" Oct 30 09:32:52.872708 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:52.872520 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f88266-2ef9-4a30-bfee-89816bc75032-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww\" (UID: \"f8f88266-2ef9-4a30-bfee-89816bc75032\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" Oct 30 09:32:52.873313 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:52.873289 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f88266-2ef9-4a30-bfee-89816bc75032-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww\" (UID: \"f8f88266-2ef9-4a30-bfee-89816bc75032\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" Oct 30 09:32:52.893626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:52.893555 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqppz\" (UniqueName: \"kubernetes.io/projected/f8f88266-2ef9-4a30-bfee-89816bc75032-kube-api-access-nqppz\") pod \"payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww\" (UID: \"f8f88266-2ef9-4a30-bfee-89816bc75032\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" Oct 30 09:32:53.046234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:53.046129 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" Oct 30 09:32:53.353390 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:53.353356 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww"] Oct 30 09:32:53.354262 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:32:53.354228 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f88266_2ef9_4a30_bfee_89816bc75032.slice/crio-0c8ef30f6a8092a8eff2fecebae3dcc9d8e7bf127e106057d56b21f31c6cdabb WatchSource:0}: Error finding container 0c8ef30f6a8092a8eff2fecebae3dcc9d8e7bf127e106057d56b21f31c6cdabb: Status 404 returned error can't find the container with id 0c8ef30f6a8092a8eff2fecebae3dcc9d8e7bf127e106057d56b21f31c6cdabb Oct 30 09:32:53.669744 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:53.669714 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" Oct 30 09:32:53.670204 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:53.669789 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" Oct 30 09:32:53.833641 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:53.830740 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" event={"ID":"f8f88266-2ef9-4a30-bfee-89816bc75032","Type":"ContainerStarted","Data":"a864682232606ce6f038dcce87a158ac0681bdaaf3e5284b9fee168064c80438"} Oct 30 09:32:53.833641 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:53.830818 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" Oct 30 09:32:53.833641 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:53.830850 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" event={"ID":"f8f88266-2ef9-4a30-bfee-89816bc75032","Type":"ContainerStarted","Data":"0c8ef30f6a8092a8eff2fecebae3dcc9d8e7bf127e106057d56b21f31c6cdabb"} Oct 30 09:32:53.874731 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:32:53.874673 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" podStartSLOduration=1.8746567870000002 podStartE2EDuration="1.874656787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:53.871726694 +0000 UTC m=+2291.734432517" watchObservedRunningTime="2026-10-30 09:32:53.874656787 +0000 UTC m=+2291.737362608" Oct 30 09:33:04.557689 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.557652 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq"] Oct 30 09:33:04.564259 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.564220 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" Oct 30 09:33:04.582663 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.582562 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq"] Oct 30 09:33:04.587842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.587815 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-aa2fdf\"" Oct 30 09:33:04.663613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.651219 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-aa2fdf-dockercfg-w85zf\"" Oct 30 09:33:04.687619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.668623 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq\" (UID: \"1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" Oct 30 09:33:04.687619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.668701 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qljf\" (UniqueName: \"kubernetes.io/projected/1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24-kube-api-access-7qljf\") pod \"payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq\" (UID: \"1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" Oct 30 09:33:04.780656 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.693683 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87"] Oct 30 09:33:04.780656 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.726031 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" Oct 30 09:33:04.780656 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.770745 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qljf\" (UniqueName: \"kubernetes.io/projected/1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24-kube-api-access-7qljf\") pod \"payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq\" (UID: \"1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" Oct 30 09:33:04.780656 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.771074 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq\" (UID: \"1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" Oct 30 09:33:04.780656 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.772159 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq\" (UID: \"1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" Oct 30 09:33:04.780656 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.779575 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87"] Oct 30 09:33:04.843462 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.843364 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" Oct 30 09:33:04.849675 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.849634 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qljf\" (UniqueName: \"kubernetes.io/projected/1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24-kube-api-access-7qljf\") pod \"payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq\" (UID: \"1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" Oct 30 09:33:04.872452 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.872409 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l9dx\" (UniqueName: \"kubernetes.io/projected/0bd2cc71-7527-48bf-a58a-db25175e2621-kube-api-access-7l9dx\") pod \"payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87\" (UID: \"0bd2cc71-7527-48bf-a58a-db25175e2621\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" Oct 30 09:33:04.872658 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.872602 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd2cc71-7527-48bf-a58a-db25175e2621-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87\" (UID: \"0bd2cc71-7527-48bf-a58a-db25175e2621\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" Oct 30 09:33:04.897929 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.897612 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" Oct 30 09:33:04.976205 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.975695 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd2cc71-7527-48bf-a58a-db25175e2621-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87\" (UID: \"0bd2cc71-7527-48bf-a58a-db25175e2621\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" Oct 30 09:33:04.976205 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.975935 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l9dx\" (UniqueName: \"kubernetes.io/projected/0bd2cc71-7527-48bf-a58a-db25175e2621-kube-api-access-7l9dx\") pod \"payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87\" (UID: \"0bd2cc71-7527-48bf-a58a-db25175e2621\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" Oct 30 09:33:04.977326 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:04.977295 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd2cc71-7527-48bf-a58a-db25175e2621-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87\" (UID: \"0bd2cc71-7527-48bf-a58a-db25175e2621\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" Oct 30 09:33:05.009074 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:05.009034 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l9dx\" (UniqueName: \"kubernetes.io/projected/0bd2cc71-7527-48bf-a58a-db25175e2621-kube-api-access-7l9dx\") pod \"payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87\" (UID: \"0bd2cc71-7527-48bf-a58a-db25175e2621\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" Oct 30 09:33:05.057924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:05.057886 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" Oct 30 09:33:05.263636 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:05.257895 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq"] Oct 30 09:33:05.265744 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:33:05.265689 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2e5d9f_2aeb_4e7f_b0f4_72e4f4afab24.slice/crio-7d03422e06d989db1844e9f9bdee57a177eb57213ebf183b166ae319f32c4dde WatchSource:0}: Error finding container 7d03422e06d989db1844e9f9bdee57a177eb57213ebf183b166ae319f32c4dde: Status 404 returned error can't find the container with id 7d03422e06d989db1844e9f9bdee57a177eb57213ebf183b166ae319f32c4dde Oct 30 09:33:05.416467 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:05.416432 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87"] Oct 30 09:33:05.417921 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:33:05.417887 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd2cc71_7527_48bf_a58a_db25175e2621.slice/crio-e7ff73af9807164d99f14889af92d4845acf5d8c55508307739c5eff5dd123ff WatchSource:0}: Error finding container e7ff73af9807164d99f14889af92d4845acf5d8c55508307739c5eff5dd123ff: Status 404 returned error can't find the container with id e7ff73af9807164d99f14889af92d4845acf5d8c55508307739c5eff5dd123ff Oct 30 09:33:06.072387 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:06.072338 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" event={"ID":"0bd2cc71-7527-48bf-a58a-db25175e2621","Type":"ContainerStarted","Data":"e752ab005a48d54d767a016ecaf3713df0512be0508bacef957b0d1578b78756"} Oct 30 09:33:06.072921 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:06.072416 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" event={"ID":"0bd2cc71-7527-48bf-a58a-db25175e2621","Type":"ContainerStarted","Data":"e7ff73af9807164d99f14889af92d4845acf5d8c55508307739c5eff5dd123ff"} Oct 30 09:33:06.072921 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:06.072783 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" Oct 30 09:33:06.075036 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:06.075002 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" event={"ID":"1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24","Type":"ContainerStarted","Data":"ba9e7c5ed8cc7393bfdcc1ceb208605fe092c6e70553aa0b93330638f77427b3"} Oct 30 09:33:06.075220 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:06.075058 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" event={"ID":"1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24","Type":"ContainerStarted","Data":"7d03422e06d989db1844e9f9bdee57a177eb57213ebf183b166ae319f32c4dde"} Oct 30 09:33:06.075220 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:06.075114 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" Oct 30 09:33:06.122603 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:06.122511 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" podStartSLOduration=2.122487049 podStartE2EDuration="2.122487049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:06.114687233 +0000 UTC m=+2303.977393058" watchObservedRunningTime="2026-10-30 09:33:06.122487049 +0000 UTC m=+2303.985192871" Oct 30 09:33:06.158737 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:06.158661 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" podStartSLOduration=2.158640156 podStartE2EDuration="2.158640156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:06.157191843 +0000 UTC m=+2304.019897665" watchObservedRunningTime="2026-10-30 09:33:06.158640156 +0000 UTC m=+2304.021345978" Oct 30 09:33:07.381820 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:07.381769 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:33:07.384394 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:07.381851 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" podUID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:33:08.094739 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:08.094631 5938 generic.go:356] "Generic (PLEG): container finished" podID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" containerID="ec34ecee1d481d600b1cd060d276523bc3f8cf8e1526f700e85fa150bf40619d" exitCode=0 Oct 30 09:33:08.094739 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:08.094712 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" event={"ID":"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe","Type":"ContainerDied","Data":"ec34ecee1d481d600b1cd060d276523bc3f8cf8e1526f700e85fa150bf40619d"} Oct 30 09:33:08.095262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:08.095242 5938 scope.go:122] "RemoveContainer" containerID="ec34ecee1d481d600b1cd060d276523bc3f8cf8e1526f700e85fa150bf40619d" Oct 30 09:33:08.097157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:08.097127 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:33:08.097274 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:08.097194 5938 generic.go:356] "Generic (PLEG): container finished" podID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" containerID="3d9a61ef06c9603c8464098f20d12f669ba8b8640bebbf1de71840e54a7aabb6" exitCode=0 Oct 30 09:33:08.097274 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:08.097212 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:33:08.097274 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:08.097245 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" event={"ID":"be2573c7-dd0a-4ba6-8e74-66cd5eef4361","Type":"ContainerDied","Data":"3d9a61ef06c9603c8464098f20d12f669ba8b8640bebbf1de71840e54a7aabb6"} Oct 30 09:33:08.097274 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:08.097253 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:33:08.097878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:08.097754 5938 scope.go:122] "RemoveContainer" containerID="3d9a61ef06c9603c8464098f20d12f669ba8b8640bebbf1de71840e54a7aabb6" Oct 30 09:33:08.100685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:08.100634 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:33:08.101074 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:08.101052 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:33:08.101163 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:08.101150 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:33:09.107012 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:09.106975 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" event={"ID":"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe","Type":"ContainerStarted","Data":"c36ecb0d6a73b3279f6399132375799c3fbbd2e13178ee037eb117550b310582"} Oct 30 09:33:09.109761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:09.109727 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" event={"ID":"be2573c7-dd0a-4ba6-8e74-66cd5eef4361","Type":"ContainerStarted","Data":"af5c573f116b3832ccbea722bf7993e9b4f182d910b2810cb8448d1d3735de65"} Oct 30 09:33:09.214614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:09.214544 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:33:09.214839 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:09.214736 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 09:33:09.214839 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:09.214783 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" podUID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 09:33:09.509753 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:09.509718 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:33:09.510072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:09.510048 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 09:33:09.510123 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:09.510095 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" podUID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 09:33:09.940892 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:09.933866 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5"] Oct 30 09:33:09.990167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:09.988251 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5"] Oct 30 09:33:09.990167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:09.988399 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" Oct 30 09:33:10.004512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.004485 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-0dc148-dockercfg-8z2vc\"" Oct 30 09:33:10.005418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.005401 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-0dc148\"" Oct 30 09:33:10.078968 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.078917 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82833df-fd04-431e-8bc2-693a47dfb44e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5\" (UID: \"c82833df-fd04-431e-8bc2-693a47dfb44e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" Oct 30 09:33:10.079171 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.079003 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b9f2\" (UniqueName: \"kubernetes.io/projected/c82833df-fd04-431e-8bc2-693a47dfb44e-kube-api-access-4b9f2\") pod \"payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5\" (UID: \"c82833df-fd04-431e-8bc2-693a47dfb44e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" Oct 30 09:33:10.180841 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.180797 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82833df-fd04-431e-8bc2-693a47dfb44e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5\" (UID: \"c82833df-fd04-431e-8bc2-693a47dfb44e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" Oct 30 09:33:10.181364 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.180867 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b9f2\" (UniqueName: \"kubernetes.io/projected/c82833df-fd04-431e-8bc2-693a47dfb44e-kube-api-access-4b9f2\") pod \"payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5\" (UID: \"c82833df-fd04-431e-8bc2-693a47dfb44e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" Oct 30 09:33:10.181850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.181826 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82833df-fd04-431e-8bc2-693a47dfb44e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5\" (UID: \"c82833df-fd04-431e-8bc2-693a47dfb44e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" Oct 30 09:33:10.201891 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.201692 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b9f2\" (UniqueName: \"kubernetes.io/projected/c82833df-fd04-431e-8bc2-693a47dfb44e-kube-api-access-4b9f2\") pod \"payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5\" (UID: \"c82833df-fd04-431e-8bc2-693a47dfb44e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" Oct 30 09:33:10.214921 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.214889 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 09:33:10.215070 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.214950 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" podUID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 09:33:10.287618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.287549 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v"] Oct 30 09:33:10.296001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.293568 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v"] Oct 30 09:33:10.296001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.293697 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:33:10.308748 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.308433 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" Oct 30 09:33:10.310553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.310520 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-304459-serving-cert\"" Oct 30 09:33:10.383849 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.383807 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04620105-1d5a-40b4-bfc3-3531870174cc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v\" (UID: \"04620105-1d5a-40b4-bfc3-3531870174cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:33:10.384030 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.383944 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04620105-1d5a-40b4-bfc3-3531870174cc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v\" (UID: \"04620105-1d5a-40b4-bfc3-3531870174cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:33:10.384226 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.384202 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04620105-1d5a-40b4-bfc3-3531870174cc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v\" (UID: \"04620105-1d5a-40b4-bfc3-3531870174cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:33:10.384750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.384727 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59xnp\" (UniqueName: \"kubernetes.io/projected/04620105-1d5a-40b4-bfc3-3531870174cc-kube-api-access-59xnp\") pod \"maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v\" (UID: \"04620105-1d5a-40b4-bfc3-3531870174cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:33:10.489926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.489708 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04620105-1d5a-40b4-bfc3-3531870174cc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v\" (UID: \"04620105-1d5a-40b4-bfc3-3531870174cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:33:10.489926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.489812 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59xnp\" (UniqueName: \"kubernetes.io/projected/04620105-1d5a-40b4-bfc3-3531870174cc-kube-api-access-59xnp\") pod \"maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v\" (UID: \"04620105-1d5a-40b4-bfc3-3531870174cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:33:10.493849 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.493802 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04620105-1d5a-40b4-bfc3-3531870174cc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v\" (UID: \"04620105-1d5a-40b4-bfc3-3531870174cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:33:10.494008 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.493902 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04620105-1d5a-40b4-bfc3-3531870174cc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v\" (UID: \"04620105-1d5a-40b4-bfc3-3531870174cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:33:10.495782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.495747 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04620105-1d5a-40b4-bfc3-3531870174cc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v\" (UID: \"04620105-1d5a-40b4-bfc3-3531870174cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:33:10.498016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.497944 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04620105-1d5a-40b4-bfc3-3531870174cc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v\" (UID: \"04620105-1d5a-40b4-bfc3-3531870174cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:33:10.500361 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.500290 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04620105-1d5a-40b4-bfc3-3531870174cc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v\" (UID: \"04620105-1d5a-40b4-bfc3-3531870174cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:33:10.572075 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.534017 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 09:33:10.572075 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.534079 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" podUID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 09:33:10.607232 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.604075 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59xnp\" (UniqueName: \"kubernetes.io/projected/04620105-1d5a-40b4-bfc3-3531870174cc-kube-api-access-59xnp\") pod \"maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v\" (UID: \"04620105-1d5a-40b4-bfc3-3531870174cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:33:10.673617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.664855 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:33:10.972838 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:10.972232 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5"] Oct 30 09:33:10.975281 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:33:10.975241 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82833df_fd04_431e_8bc2_693a47dfb44e.slice/crio-ba14d7d093486bc281cafd444c231cb690ffe86cf45abf971502d5c4d25655f3 WatchSource:0}: Error finding container ba14d7d093486bc281cafd444c231cb690ffe86cf45abf971502d5c4d25655f3: Status 404 returned error can't find the container with id ba14d7d093486bc281cafd444c231cb690ffe86cf45abf971502d5c4d25655f3 Oct 30 09:33:11.058685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:11.055333 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v"] Oct 30 09:33:11.089350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:11.086573 5938 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:33:11.216543 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:11.216005 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 09:33:11.216543 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:11.216087 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" podUID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 09:33:11.229615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:11.227991 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" event={"ID":"04620105-1d5a-40b4-bfc3-3531870174cc","Type":"ContainerStarted","Data":"5fa53d4adc9d8302db0d50488ebac81ffac1df6c729ed3c5b6a5449c855c7fcb"} Oct 30 09:33:11.260551 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:11.260377 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" event={"ID":"c82833df-fd04-431e-8bc2-693a47dfb44e","Type":"ContainerStarted","Data":"ba14d7d093486bc281cafd444c231cb690ffe86cf45abf971502d5c4d25655f3"} Oct 30 09:33:11.581612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:11.536130 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 09:33:11.585184 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:11.585124 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" podUID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 09:33:12.216642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:12.215984 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 09:33:12.216642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:12.216055 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" podUID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 09:33:12.315558 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:12.315516 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" event={"ID":"04620105-1d5a-40b4-bfc3-3531870174cc","Type":"ContainerStarted","Data":"bee4fc18f0befe6c4cefa2902878287fad3d4c5b360c35e91b8981184ecfd6c3"} Oct 30 09:33:12.316623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:12.316602 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:33:12.343614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:12.341945 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" event={"ID":"c82833df-fd04-431e-8bc2-693a47dfb44e","Type":"ContainerStarted","Data":"6c4589611a6b80b1657e1d7efa51238d5187c448e4a4872336f6709465fc19d7"} Oct 30 09:33:12.343614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:12.342935 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" Oct 30 09:33:12.418153 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:12.418039 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" podStartSLOduration=3.417974103 podStartE2EDuration="3.417974103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:12.416926536 +0000 UTC m=+2310.279632370" watchObservedRunningTime="2026-10-30 09:33:12.417974103 +0000 UTC m=+2310.280679925" Oct 30 09:33:12.457613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:12.453619 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" podStartSLOduration=1.859244744 podStartE2EDuration="2.45357281s" totalImagesPullingTime="594.328066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:12.376858197 +0000 UTC m=+2310.239564027" watchObservedRunningTime="2026-10-30 09:33:12.45357281 +0000 UTC m=+2310.316278634" Oct 30 09:33:12.510112 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:12.510009 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 09:33:12.510112 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:12.510065 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" podUID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 09:33:13.214976 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:13.214934 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 09:33:13.215212 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:13.215012 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" podUID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 09:33:13.509767 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:13.509650 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 09:33:13.509767 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:13.509742 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" podUID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 09:33:14.215160 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:14.215124 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 09:33:14.215497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:14.215465 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" podUID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 09:33:14.509216 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:14.509129 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 09:33:14.509452 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:14.509427 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" podUID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 09:33:15.219680 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:15.219646 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:33:15.220140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:15.220115 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:33:15.220985 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:15.220964 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:33:15.515149 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:15.515051 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:33:15.515484 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:15.515456 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:33:15.516925 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:15.516903 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:33:15.811738 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:33:15.811651 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:33:17.099617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:17.099261 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" Oct 30 09:33:17.101244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:17.101218 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" Oct 30 09:33:18.587277 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:18.587240 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28"] Oct 30 09:33:18.587820 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:18.587496 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" podUID="c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" containerName="istio-proxy" containerID="cri-o://6e8cecce79859d2544d3cefdf645c96d98b92200c52b9df01403c45033c73214" gracePeriod=30 Oct 30 09:33:19.384342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:19.384311 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:33:20.065214 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:20.065179 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:33:20.065724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:20.065243 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" podUID="c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:33:23.875702 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.875669 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:33:23.961394 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.961352 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-workload-socket\") pod \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " Oct 30 09:33:23.961610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.961430 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-kube-api-access-pnqqw\" (UniqueName: \"kubernetes.io/projected/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-kube-api-access-pnqqw\") pod \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " Oct 30 09:33:23.961610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.961489 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-workload-certs\") pod \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " Oct 30 09:33:23.961610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.961563 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istiod-ca-cert\") pod \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " Oct 30 09:33:23.961805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.961619 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-token\" (UniqueName: \"kubernetes.io/projected/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-token\") pod \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " Oct 30 09:33:23.961805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.961647 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-workload-socket" pod "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" (UID: "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:33:23.961805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.961715 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-envoy\") pod \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " Oct 30 09:33:23.961956 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.961812 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-workload-certs" pod "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" (UID: "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:33:23.961956 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.961874 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-data\") pod \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " Oct 30 09:33:23.961956 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.961918 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-podinfo\") pod \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " Oct 30 09:33:23.962092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.961976 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istiod-ca-cert" pod "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" (UID: "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:33:23.962092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.962002 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-credential-socket\") pod \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\" (UID: \"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb\") " Oct 30 09:33:23.962189 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.962106 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-data" pod "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" (UID: "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:33:23.962277 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.962253 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-credential-socket" pod "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" (UID: "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:33:23.962852 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.962831 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:33:23.962936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.962857 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:33:23.962936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.962873 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:33:23.962936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.962887 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:33:23.962936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.962900 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:33:23.964318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.964289 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-podinfo" pod "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" (UID: "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:33:23.964683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.964657 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-token" pod "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" (UID: "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:33:23.964864 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.964843 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-envoy" pod "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" (UID: "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:33:23.964958 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:23.964938 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-kube-api-access-pnqqw" pod "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" (UID: "c22f8b1f-a12d-4031-98eb-8acb7b00c1fb"). InnerVolumeSpecName "kube-api-access-pnqqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:33:24.064485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:24.064442 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:33:24.064485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:24.064478 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnqqw\" (UniqueName: \"kubernetes.io/projected/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-kube-api-access-pnqqw\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:33:24.064485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:24.064489 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:33:24.064485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:24.064497 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:33:24.381697 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:24.381667 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" Oct 30 09:33:24.470115 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:24.470077 5938 generic.go:356] "Generic (PLEG): container finished" podID="c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" containerID="6e8cecce79859d2544d3cefdf645c96d98b92200c52b9df01403c45033c73214" exitCode=0 Oct 30 09:33:24.470343 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:24.470137 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" Oct 30 09:33:24.470343 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:24.470158 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" event={"ID":"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb","Type":"ContainerDied","Data":"6e8cecce79859d2544d3cefdf645c96d98b92200c52b9df01403c45033c73214"} Oct 30 09:33:24.470343 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:24.470223 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28" event={"ID":"c22f8b1f-a12d-4031-98eb-8acb7b00c1fb","Type":"ContainerDied","Data":"0facc7ab00a999067a4150863819566304b1f64d94db789876920a73c767e613"} Oct 30 09:33:24.470343 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:24.470248 5938 scope.go:122] "RemoveContainer" containerID="6e8cecce79859d2544d3cefdf645c96d98b92200c52b9df01403c45033c73214" Oct 30 09:33:24.484692 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:24.484666 5938 scope.go:122] "RemoveContainer" containerID="6e8cecce79859d2544d3cefdf645c96d98b92200c52b9df01403c45033c73214" Oct 30 09:33:24.485070 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:33:24.485047 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8cecce79859d2544d3cefdf645c96d98b92200c52b9df01403c45033c73214\": container with ID starting with 6e8cecce79859d2544d3cefdf645c96d98b92200c52b9df01403c45033c73214 not found: ID does not exist" containerID="6e8cecce79859d2544d3cefdf645c96d98b92200c52b9df01403c45033c73214" Oct 30 09:33:24.485147 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:24.485081 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8cecce79859d2544d3cefdf645c96d98b92200c52b9df01403c45033c73214"} err="failed to get container status \"6e8cecce79859d2544d3cefdf645c96d98b92200c52b9df01403c45033c73214\": rpc error: code = NotFound desc = could not find container \"6e8cecce79859d2544d3cefdf645c96d98b92200c52b9df01403c45033c73214\": container with ID starting with 6e8cecce79859d2544d3cefdf645c96d98b92200c52b9df01403c45033c73214 not found: ID does not exist" Oct 30 09:33:24.529233 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:24.529143 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28"] Oct 30 09:33:24.542898 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:24.542858 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-48d28"] Oct 30 09:33:26.259023 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:26.258986 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" path="/var/lib/kubelet/pods/c22f8b1f-a12d-4031-98eb-8acb7b00c1fb/volumes" Oct 30 09:33:30.142880 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.142836 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c"] Oct 30 09:33:30.143725 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.143704 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" containerName="istio-proxy" Oct 30 09:33:30.143857 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.143728 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" containerName="istio-proxy" Oct 30 09:33:30.143958 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.143943 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c22f8b1f-a12d-4031-98eb-8acb7b00c1fb" containerName="istio-proxy" Oct 30 09:33:30.149473 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.149442 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.169990 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.169950 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c"] Oct 30 09:33:30.177483 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.177450 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-19c94a\"/\"kube-root-ca.crt\"" Oct 30 09:33:30.177483 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.177450 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-19c94a\"/\"openshift-service-ca.crt\"" Oct 30 09:33:30.177754 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.177629 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-19c94a\"/\"auth-test-model-19c94a-kserve-self-signed-certs\"" Oct 30 09:33:30.181927 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.181899 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-19c94a\"/\"default-dockercfg-4jvn8\"" Oct 30 09:33:30.262240 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.262205 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-home\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.262455 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.262407 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-tmp-dir\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.262548 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.262532 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/957b9998-b762-4673-bf29-1d5b8d7ae904-tls-certs\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.262666 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.262648 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4r26\" (UniqueName: \"kubernetes.io/projected/957b9998-b762-4673-bf29-1d5b8d7ae904-kube-api-access-w4r26\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.262760 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.262740 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-model-cache\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.262823 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.262805 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-dshm\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.369628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.369513 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-tmp-dir\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.369628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.369643 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/957b9998-b762-4673-bf29-1d5b8d7ae904-tls-certs\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.370216 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.369704 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4r26\" (UniqueName: \"kubernetes.io/projected/957b9998-b762-4673-bf29-1d5b8d7ae904-kube-api-access-w4r26\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.370216 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.369731 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-model-cache\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.370216 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.369783 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-dshm\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.370216 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.369930 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-home\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.377295 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.370795 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-tmp-dir\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.377295 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.371813 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-model-cache\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.377295 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.372317 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-home\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.403421 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.402290 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/957b9998-b762-4673-bf29-1d5b8d7ae904-tls-certs\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.412614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.411232 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-dshm\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.416617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.415863 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4r26\" (UniqueName: \"kubernetes.io/projected/957b9998-b762-4673-bf29-1d5b8d7ae904-kube-api-access-w4r26\") pod \"auth-test-model-19c94a-kserve-744b56c457-qgl2c\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.473411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.472075 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:30.820116 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:30.820074 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c"] Oct 30 09:33:30.820431 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:33:30.820404 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod957b9998_b762_4673_bf29_1d5b8d7ae904.slice/crio-2b29a7b38f0dc73e113b5421ede3398e188e867ee52cb459e0afb34798840577 WatchSource:0}: Error finding container 2b29a7b38f0dc73e113b5421ede3398e188e867ee52cb459e0afb34798840577: Status 404 returned error can't find the container with id 2b29a7b38f0dc73e113b5421ede3398e188e867ee52cb459e0afb34798840577 Oct 30 09:33:31.552050 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:31.552004 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" event={"ID":"957b9998-b762-4673-bf29-1d5b8d7ae904","Type":"ContainerStarted","Data":"0c0f5c22f720518e8f00462ed57f9bfedf665d73cdc62e506cb50b51ab7f3b9b"} Oct 30 09:33:31.552485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:31.552063 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" event={"ID":"957b9998-b762-4673-bf29-1d5b8d7ae904","Type":"ContainerStarted","Data":"2b29a7b38f0dc73e113b5421ede3398e188e867ee52cb459e0afb34798840577"} Oct 30 09:33:31.596000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:31.595925 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" podStartSLOduration=1.595910355 podStartE2EDuration="1.595910355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:31.592533317 +0000 UTC m=+2329.455239138" watchObservedRunningTime="2026-10-30 09:33:31.595910355 +0000 UTC m=+2329.458616215" Oct 30 09:33:32.472569 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:32.472523 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:40.473077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:40.473032 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:40.486057 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:40.486024 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:40.643913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:40.643882 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:33:43.053310 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:43.053276 5938 scope.go:122] "RemoveContainer" containerID="1fd26f6d84ef39a7afd55d671189cc20f5fe5be3b3b1b2e7e9dac5b56e2b167a" Oct 30 09:33:45.820145 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:33:45.820104 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:33:46.806747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.806709 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx"] Oct 30 09:33:46.813202 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.813176 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.834635 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.834569 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx"] Oct 30 09:33:46.843211 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.843165 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-304459-openshift-default-dockercfg-8w4bn\"" Oct 30 09:33:46.864535 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.864499 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.864768 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.864641 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj9vc\" (UniqueName: \"kubernetes.io/projected/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-kube-api-access-fj9vc\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.864768 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.864736 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.864862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.864794 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.864862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.864815 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.864946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.864933 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.864990 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.864978 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.865032 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.865006 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.865083 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.865043 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.966188 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.966145 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.966188 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.966195 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.966481 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.966246 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.966481 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.966422 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.966481 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.966477 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.966715 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.966516 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.966715 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.966572 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.966838 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.966773 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.966838 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.966791 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.966952 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.966869 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj9vc\" (UniqueName: \"kubernetes.io/projected/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-kube-api-access-fj9vc\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.966952 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.966890 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.966952 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.966936 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.967157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.967132 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.967222 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.967173 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.969084 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.969056 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.969196 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.969130 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.987424 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.987388 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:46.987628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:46.987573 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj9vc\" (UniqueName: \"kubernetes.io/projected/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-kube-api-access-fj9vc\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:47.130299 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:47.130251 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:47.489573 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:47.488216 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx"] Oct 30 09:33:47.499906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:47.499862 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:33:47.500189 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:47.500171 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:33:47.500284 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:47.500273 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:33:47.685884 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:47.685841 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" event={"ID":"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f","Type":"ContainerStarted","Data":"8d7ae38d765acac1fc64316d785e15986dbc83a464f859090b179d1091ed2c2c"} Oct 30 09:33:47.686097 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:47.685916 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" event={"ID":"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f","Type":"ContainerStarted","Data":"50e18671e1b38bfadbb8ef1749d6f9c582d30561bc934a1afb37b225f3e8a2ef"} Oct 30 09:33:47.734490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:47.734296 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" podStartSLOduration=1.734271208 podStartE2EDuration="1.734271208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:47.731978469 +0000 UTC m=+2345.594684295" watchObservedRunningTime="2026-10-30 09:33:47.734271208 +0000 UTC m=+2345.596977045" Oct 30 09:33:48.136608 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:48.135650 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:48.145623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:48.142916 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:33:48.145623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:48.142980 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" podUID="aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 09:33:49.134616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.131068 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:33:49.134616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.131139 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" podUID="aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 09:33:49.713423 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.713382 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns"] Oct 30 09:33:49.745614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.719319 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.763966 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.763936 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns"] Oct 30 09:33:49.767316 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.767287 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-7c88ae\"/\"kube-root-ca.crt\"" Oct 30 09:33:49.767476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.767352 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-7c88ae\"/\"default-dockercfg-dm6gn\"" Oct 30 09:33:49.767476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.767405 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-7c88ae\"/\"openshift-service-ca.crt\"" Oct 30 09:33:49.767476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.767458 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-7c88ae\"/\"auth-test-model-7c88ae-kserve-self-signed-certs\"" Oct 30 09:33:49.815369 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.815316 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-model-cache\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.815369 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.815370 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-home\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.815738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.815457 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-dshm\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.815738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.815559 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/822b48c6-2e86-45a3-af80-ef8cd9cf5674-tls-certs\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.815738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.815699 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8546\" (UniqueName: \"kubernetes.io/projected/822b48c6-2e86-45a3-af80-ef8cd9cf5674-kube-api-access-q8546\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.815738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.815728 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-tmp-dir\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.917134 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.917067 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/822b48c6-2e86-45a3-af80-ef8cd9cf5674-tls-certs\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.917334 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.917225 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8546\" (UniqueName: \"kubernetes.io/projected/822b48c6-2e86-45a3-af80-ef8cd9cf5674-kube-api-access-q8546\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.917334 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.917272 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-tmp-dir\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.917444 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.917370 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-model-cache\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.917444 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.917399 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-home\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.917547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.917495 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-dshm\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.918051 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.917971 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-tmp-dir\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.918564 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.918271 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-model-cache\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.918564 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.918513 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-home\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.925031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.921223 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/822b48c6-2e86-45a3-af80-ef8cd9cf5674-tls-certs\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.942611 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.932201 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-dshm\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:49.978671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:49.972701 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8546\" (UniqueName: \"kubernetes.io/projected/822b48c6-2e86-45a3-af80-ef8cd9cf5674-kube-api-access-q8546\") pod \"auth-test-model-7c88ae-kserve-8898c9765-pxjns\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:50.082710 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:50.081700 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:50.141646 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:50.139120 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:33:50.141646 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:50.139200 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" podUID="aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 09:33:50.470296 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:50.470262 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns"] Oct 30 09:33:50.713582 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:50.713543 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" event={"ID":"822b48c6-2e86-45a3-af80-ef8cd9cf5674","Type":"ContainerStarted","Data":"3ba907c858e6fa8cc1dea62b5656d6e9c65975a4872d8797b17efff5590da984"} Oct 30 09:33:50.714110 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:50.713621 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" event={"ID":"822b48c6-2e86-45a3-af80-ef8cd9cf5674","Type":"ContainerStarted","Data":"1fffbbca89f0dce3cbc16fe414acfb35a848fa24eb308ef4ff736e83656f1ef8"} Oct 30 09:33:50.762330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:50.762214 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" podStartSLOduration=1.762186066 podStartE2EDuration="1.762186066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:50.758214716 +0000 UTC m=+2348.620920555" watchObservedRunningTime="2026-10-30 09:33:50.762186066 +0000 UTC m=+2348.624891887" Oct 30 09:33:51.083837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:51.083706 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:33:51.134178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:51.132073 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:33:51.134178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:51.132147 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" podUID="aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 09:33:52.135678 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:52.134047 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:33:52.135678 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:52.134111 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" podUID="aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 09:33:53.143617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:53.141042 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:33:53.143617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:53.141107 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" podUID="aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 09:33:54.135644 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:54.135607 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:54.136121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:54.136100 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:33:54.136910 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:33:54.136889 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:34:00.083843 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:00.083795 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:34:00.103879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:00.103845 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:34:00.882391 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:00.882363 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:34:08.594506 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:08.594474 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc"] Oct 30 09:34:08.595031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:08.594709 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" podUID="7a2b8ea1-4db6-479d-b681-a9a26b02ddec" containerName="istio-proxy" containerID="cri-o://558ab888bd950576602ca02dab19593da8d34ca1cd3e111e0fecc01a606fc631" gracePeriod=30 Oct 30 09:34:13.366151 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.366108 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr"] Oct 30 09:34:13.370927 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.370899 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.402482 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.402389 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-fc7ee7\"/\"default-dockercfg-m7k4r\"" Oct 30 09:34:13.402880 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.402860 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-fc7ee7\"/\"kube-root-ca.crt\"" Oct 30 09:34:13.403140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.403122 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-fc7ee7\"/\"auth-test-model-fc7ee7-kserve-self-signed-certs\"" Oct 30 09:34:13.403817 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.403781 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-fc7ee7\"/\"openshift-service-ca.crt\"" Oct 30 09:34:13.409756 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.409723 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr"] Oct 30 09:34:13.481404 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.481363 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzxzz\" (UniqueName: \"kubernetes.io/projected/64267c50-7a21-4c55-aa19-f0899841dad6-kube-api-access-zzxzz\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.481616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.481539 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-tmp-dir\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.481672 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.481580 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-home\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.481835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.481810 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64267c50-7a21-4c55-aa19-f0899841dad6-tls-certs\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.481886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.481860 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-dshm\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.481951 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.481931 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-model-cache\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.585816 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.585528 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64267c50-7a21-4c55-aa19-f0899841dad6-tls-certs\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.585816 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.585623 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-dshm\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.585816 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.585657 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-model-cache\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.586146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.585837 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzxzz\" (UniqueName: \"kubernetes.io/projected/64267c50-7a21-4c55-aa19-f0899841dad6-kube-api-access-zzxzz\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.586146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.585950 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-tmp-dir\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.586146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.585984 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-home\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.589580 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.586550 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-home\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.589580 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.586862 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-tmp-dir\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.589580 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.586955 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-model-cache\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.613522 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.613433 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64267c50-7a21-4c55-aa19-f0899841dad6-tls-certs\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.623717 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.623575 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzxzz\" (UniqueName: \"kubernetes.io/projected/64267c50-7a21-4c55-aa19-f0899841dad6-kube-api-access-zzxzz\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.623914 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.623891 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-dshm\") pod \"auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.688655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.688115 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:13.921493 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.921464 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:34:13.992089 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.992059 5938 generic.go:356] "Generic (PLEG): container finished" podID="7a2b8ea1-4db6-479d-b681-a9a26b02ddec" containerID="558ab888bd950576602ca02dab19593da8d34ca1cd3e111e0fecc01a606fc631" exitCode=0 Oct 30 09:34:13.992305 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.992121 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" event={"ID":"7a2b8ea1-4db6-479d-b681-a9a26b02ddec","Type":"ContainerDied","Data":"558ab888bd950576602ca02dab19593da8d34ca1cd3e111e0fecc01a606fc631"} Oct 30 09:34:13.992305 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.992196 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" event={"ID":"7a2b8ea1-4db6-479d-b681-a9a26b02ddec","Type":"ContainerDied","Data":"4819277cbd40df0d6383de2a5632beb3a8faf18cb3e5e37f40166c64b4791034"} Oct 30 09:34:13.992305 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.992131 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc" Oct 30 09:34:13.992435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.992230 5938 scope.go:122] "RemoveContainer" containerID="558ab888bd950576602ca02dab19593da8d34ca1cd3e111e0fecc01a606fc631" Oct 30 09:34:13.993758 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.993738 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-workload-certs\") pod \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " Oct 30 09:34:13.993859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.993827 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-token\" (UniqueName: \"kubernetes.io/projected/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-token\") pod \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " Oct 30 09:34:13.993924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.993900 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-data\") pod \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " Oct 30 09:34:13.993982 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.993955 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-kube-api-access-9549w\" (UniqueName: \"kubernetes.io/projected/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-kube-api-access-9549w\") pod \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " Oct 30 09:34:13.994036 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.994013 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-podinfo\") pod \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " Oct 30 09:34:13.994231 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.994114 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-workload-certs" pod "7a2b8ea1-4db6-479d-b681-a9a26b02ddec" (UID: "7a2b8ea1-4db6-479d-b681-a9a26b02ddec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:13.994231 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.994122 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-workload-socket\") pod \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " Oct 30 09:34:13.994231 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.994203 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-credential-socket\") pod \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " Oct 30 09:34:13.994412 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.994245 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istiod-ca-cert\") pod \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " Oct 30 09:34:13.994412 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.994373 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-envoy\") pod \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\" (UID: \"7a2b8ea1-4db6-479d-b681-a9a26b02ddec\") " Oct 30 09:34:13.994830 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.994806 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-data" pod "7a2b8ea1-4db6-479d-b681-a9a26b02ddec" (UID: "7a2b8ea1-4db6-479d-b681-a9a26b02ddec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:13.995260 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.995240 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:13.995390 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.995376 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:13.995511 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.995456 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-credential-socket" pod "7a2b8ea1-4db6-479d-b681-a9a26b02ddec" (UID: "7a2b8ea1-4db6-479d-b681-a9a26b02ddec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:13.995738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.995694 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istiod-ca-cert" pod "7a2b8ea1-4db6-479d-b681-a9a26b02ddec" (UID: "7a2b8ea1-4db6-479d-b681-a9a26b02ddec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:34:13.996149 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.996124 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-workload-socket" pod "7a2b8ea1-4db6-479d-b681-a9a26b02ddec" (UID: "7a2b8ea1-4db6-479d-b681-a9a26b02ddec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:13.998451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.997553 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-envoy" pod "7a2b8ea1-4db6-479d-b681-a9a26b02ddec" (UID: "7a2b8ea1-4db6-479d-b681-a9a26b02ddec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:13.998451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.997657 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-kube-api-access-9549w" pod "7a2b8ea1-4db6-479d-b681-a9a26b02ddec" (UID: "7a2b8ea1-4db6-479d-b681-a9a26b02ddec"). InnerVolumeSpecName "kube-api-access-9549w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:34:13.998451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.997752 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-token" pod "7a2b8ea1-4db6-479d-b681-a9a26b02ddec" (UID: "7a2b8ea1-4db6-479d-b681-a9a26b02ddec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:34:14.001630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:13.999438 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-podinfo" pod "7a2b8ea1-4db6-479d-b681-a9a26b02ddec" (UID: "7a2b8ea1-4db6-479d-b681-a9a26b02ddec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:34:14.006139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:14.006107 5938 scope.go:122] "RemoveContainer" containerID="558ab888bd950576602ca02dab19593da8d34ca1cd3e111e0fecc01a606fc631" Oct 30 09:34:14.007056 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:34:14.006723 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"558ab888bd950576602ca02dab19593da8d34ca1cd3e111e0fecc01a606fc631\": container with ID starting with 558ab888bd950576602ca02dab19593da8d34ca1cd3e111e0fecc01a606fc631 not found: ID does not exist" containerID="558ab888bd950576602ca02dab19593da8d34ca1cd3e111e0fecc01a606fc631" Oct 30 09:34:14.007056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:14.006773 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"558ab888bd950576602ca02dab19593da8d34ca1cd3e111e0fecc01a606fc631"} err="failed to get container status \"558ab888bd950576602ca02dab19593da8d34ca1cd3e111e0fecc01a606fc631\": rpc error: code = NotFound desc = could not find container \"558ab888bd950576602ca02dab19593da8d34ca1cd3e111e0fecc01a606fc631\": container with ID starting with 558ab888bd950576602ca02dab19593da8d34ca1cd3e111e0fecc01a606fc631 not found: ID does not exist" Oct 30 09:34:14.022694 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:14.022663 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr"] Oct 30 09:34:14.023763 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:34:14.023725 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64267c50_7a21_4c55_aa19_f0899841dad6.slice/crio-fd8129d6960634fadcf548b675e3526c6c7bd8b95f1483ff5ded9c2e5de71422 WatchSource:0}: Error finding container fd8129d6960634fadcf548b675e3526c6c7bd8b95f1483ff5ded9c2e5de71422: Status 404 returned error can't find the container with id fd8129d6960634fadcf548b675e3526c6c7bd8b95f1483ff5ded9c2e5de71422 Oct 30 09:34:14.096550 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:14.096512 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:14.096550 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:14.096545 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:14.096550 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:14.096554 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9549w\" (UniqueName: \"kubernetes.io/projected/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-kube-api-access-9549w\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:14.096878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:14.096563 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:14.096878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:14.096574 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:14.096878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:14.096599 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:14.096878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:14.096613 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2b8ea1-4db6-479d-b681-a9a26b02ddec-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:14.342297 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:14.342255 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc"] Oct 30 09:34:14.347095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:14.347062 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-npnjc"] Oct 30 09:34:15.003008 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:15.002964 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" event={"ID":"64267c50-7a21-4c55-aa19-f0899841dad6","Type":"ContainerStarted","Data":"c1b1bef1de7824e938d771ae100f479e7e84c361a3c8160416f3c6850d18eaaa"} Oct 30 09:34:15.003435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:15.003027 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" event={"ID":"64267c50-7a21-4c55-aa19-f0899841dad6","Type":"ContainerStarted","Data":"fd8129d6960634fadcf548b675e3526c6c7bd8b95f1483ff5ded9c2e5de71422"} Oct 30 09:34:15.045253 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:15.045186 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" podStartSLOduration=2.045168893 podStartE2EDuration="2.045168893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:34:15.043838997 +0000 UTC m=+2372.906544821" watchObservedRunningTime="2026-10-30 09:34:15.045168893 +0000 UTC m=+2372.907874713" Oct 30 09:34:15.688644 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:15.688563 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:15.811054 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:34:15.811014 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:34:16.259022 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.258987 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2b8ea1-4db6-479d-b681-a9a26b02ddec" path="/var/lib/kubelet/pods/7a2b8ea1-4db6-479d-b681-a9a26b02ddec/volumes" Oct 30 09:34:16.567083 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.567009 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8"] Oct 30 09:34:16.567666 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.567650 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7a2b8ea1-4db6-479d-b681-a9a26b02ddec" containerName="istio-proxy" Oct 30 09:34:16.567732 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.567668 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2b8ea1-4db6-479d-b681-a9a26b02ddec" containerName="istio-proxy" Oct 30 09:34:16.567813 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.567795 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2b8ea1-4db6-479d-b681-a9a26b02ddec" containerName="istio-proxy" Oct 30 09:34:16.571985 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.571958 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.602496 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.602453 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8"] Oct 30 09:34:16.607668 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.607636 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-fb1a2a\"/\"kube-root-ca.crt\"" Oct 30 09:34:16.607911 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.607672 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-fb1a2a\"/\"auth-test-model-fb1a2a-kserve-self-signed-certs\"" Oct 30 09:34:16.608004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.607741 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-fb1a2a\"/\"default-dockercfg-lg86p\"" Oct 30 09:34:16.608004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.607821 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-fb1a2a\"/\"openshift-service-ca.crt\"" Oct 30 09:34:16.634182 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.634142 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-model-cache\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.634418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.634216 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-tmp-dir\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.634418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.634302 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-home\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.634418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.634346 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-dshm\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.634418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.634399 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7zzl\" (UniqueName: \"kubernetes.io/projected/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-kube-api-access-w7zzl\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.634813 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.634537 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-tls-certs\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.735867 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.735830 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-tls-certs\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.736048 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.735918 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-model-cache\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.736048 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.735953 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-tmp-dir\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.736048 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.736013 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-home\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.736048 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.736040 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-dshm\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.736284 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.736073 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7zzl\" (UniqueName: \"kubernetes.io/projected/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-kube-api-access-w7zzl\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.736530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.736496 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-model-cache\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.736674 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.736515 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-home\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.736782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.736760 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-tmp-dir\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.738816 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.738792 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-tls-certs\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.739004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.738983 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-dshm\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.757037 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.757004 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7zzl\" (UniqueName: \"kubernetes.io/projected/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-kube-api-access-w7zzl\") pod \"auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:16.897833 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:16.897797 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:17.232334 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:34:17.232286 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6296acea_d4ae_4735_b8ab_22f8ed8d2bd8.slice/crio-4f9046906d666bed4677c35f5bc0ad3bddfa01cfd02c0350bd49ef56af378890 WatchSource:0}: Error finding container 4f9046906d666bed4677c35f5bc0ad3bddfa01cfd02c0350bd49ef56af378890: Status 404 returned error can't find the container with id 4f9046906d666bed4677c35f5bc0ad3bddfa01cfd02c0350bd49ef56af378890 Oct 30 09:34:17.232680 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:17.232656 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8"] Oct 30 09:34:18.033762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:18.033725 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" event={"ID":"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8","Type":"ContainerStarted","Data":"4b872b584359fc8741c0c85c4fe173d08d74e99f298c8718bcf0c6caab53434e"} Oct 30 09:34:18.033762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:18.033773 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" event={"ID":"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8","Type":"ContainerStarted","Data":"4f9046906d666bed4677c35f5bc0ad3bddfa01cfd02c0350bd49ef56af378890"} Oct 30 09:34:18.081358 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:18.081300 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" podStartSLOduration=2.081284176 podStartE2EDuration="2.081284176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:34:18.078669108 +0000 UTC m=+2375.941374929" watchObservedRunningTime="2026-10-30 09:34:18.081284176 +0000 UTC m=+2375.943990070" Oct 30 09:34:18.898148 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:18.898102 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:23.037225 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:23.037175 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7"] Oct 30 09:34:23.041303 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:23.041272 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d639a65f-f6bx7"] Oct 30 09:34:23.688751 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:23.688716 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:23.702259 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:23.702231 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:24.097853 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:24.097753 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:34:24.262670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:24.262614 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7631eda-702a-4552-9fa4-cae2275317d3" path="/var/lib/kubelet/pods/e7631eda-702a-4552-9fa4-cae2275317d3/volumes" Oct 30 09:34:26.899124 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:26.899063 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:26.927216 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:26.927185 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:27.118706 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:27.118669 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:34:31.997121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:31.997078 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv"] Oct 30 09:34:32.003151 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.003123 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.039495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.039458 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv"] Oct 30 09:34:32.044995 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.044961 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj"] Oct 30 09:34:32.051414 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.051376 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba44b3e-lbxrj"] Oct 30 09:34:32.125613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.118623 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.125613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.118812 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.125613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.118886 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80566eca-1f41-450c-968e-f917a76cc203-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.125613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.118962 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80566eca-1f41-450c-968e-f917a76cc203-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.125613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.118989 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4bnw\" (UniqueName: \"kubernetes.io/projected/80566eca-1f41-450c-968e-f917a76cc203-kube-api-access-v4bnw\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.125613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.119046 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80566eca-1f41-450c-968e-f917a76cc203-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.125613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.119134 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.125613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.119346 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.125613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.119382 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.224364 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.224309 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.224364 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.224366 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.224683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.224487 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.224683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.224581 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.224683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.224659 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80566eca-1f41-450c-968e-f917a76cc203-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.224868 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.224721 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80566eca-1f41-450c-968e-f917a76cc203-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.224868 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.224746 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4bnw\" (UniqueName: \"kubernetes.io/projected/80566eca-1f41-450c-968e-f917a76cc203-kube-api-access-v4bnw\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.224868 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.224796 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80566eca-1f41-450c-968e-f917a76cc203-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.224868 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.224838 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.224868 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.224852 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.225081 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.224897 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.225081 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.224950 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.225299 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.225184 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.225964 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.225917 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80566eca-1f41-450c-968e-f917a76cc203-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.228334 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.228310 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.228499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.228438 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80566eca-1f41-450c-968e-f917a76cc203-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.250487 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.250394 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80566eca-1f41-450c-968e-f917a76cc203-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.250487 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.250474 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4bnw\" (UniqueName: \"kubernetes.io/projected/80566eca-1f41-450c-968e-f917a76cc203-kube-api-access-v4bnw\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.264614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.263941 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd86d9a-04a2-40f3-b74c-2b92781a0b59" path="/var/lib/kubelet/pods/ddd86d9a-04a2-40f3-b74c-2b92781a0b59/volumes" Oct 30 09:34:32.336922 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.336879 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:32.764000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.763957 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv"] Oct 30 09:34:32.765218 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:34:32.765192 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80566eca_1f41_450c_968e_f917a76cc203.slice/crio-5878460f70d6dfb49935c9ef56c48a1e85debf9df235598511b14ac998eb21cb WatchSource:0}: Error finding container 5878460f70d6dfb49935c9ef56c48a1e85debf9df235598511b14ac998eb21cb: Status 404 returned error can't find the container with id 5878460f70d6dfb49935c9ef56c48a1e85debf9df235598511b14ac998eb21cb Oct 30 09:34:32.768257 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.768221 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:34:32.768371 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.768325 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:34:32.768439 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:32.768374 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:34:33.181030 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:33.180975 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" event={"ID":"80566eca-1f41-450c-968e-f917a76cc203","Type":"ContainerStarted","Data":"30c18541d9eef9767ebae746842723c6b55110bbb8ddb7f257e4b44a37a1b7d8"} Oct 30 09:34:33.181472 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:33.181054 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" event={"ID":"80566eca-1f41-450c-968e-f917a76cc203","Type":"ContainerStarted","Data":"5878460f70d6dfb49935c9ef56c48a1e85debf9df235598511b14ac998eb21cb"} Oct 30 09:34:33.226039 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:33.225968 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" podStartSLOduration=2.225944919 podStartE2EDuration="2.225944919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:34:33.223735354 +0000 UTC m=+2391.086441175" watchObservedRunningTime="2026-10-30 09:34:33.225944919 +0000 UTC m=+2391.088650741" Oct 30 09:34:33.341406 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:33.340006 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:33.345613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:33.343727 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 09:34:33.345613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:33.343820 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" podUID="80566eca-1f41-450c-968e-f917a76cc203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 09:34:34.338628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:34.338072 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 09:34:34.338628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:34.338144 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" podUID="80566eca-1f41-450c-968e-f917a76cc203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 09:34:35.340670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:35.339044 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 09:34:35.340670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:35.339116 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" podUID="80566eca-1f41-450c-968e-f917a76cc203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 09:34:36.046613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:36.046548 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz"] Oct 30 09:34:36.051404 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:36.051368 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db5c9dd-bhzfz"] Oct 30 09:34:36.259102 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:36.259068 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0031b941-1629-4bc1-81b6-c095ae05d6f5" path="/var/lib/kubelet/pods/0031b941-1629-4bc1-81b6-c095ae05d6f5/volumes" Oct 30 09:34:36.342149 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:36.342051 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 09:34:36.342777 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:36.342738 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" podUID="80566eca-1f41-450c-968e-f917a76cc203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 09:34:37.340205 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:37.339697 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 09:34:37.340205 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:37.339761 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" podUID="80566eca-1f41-450c-968e-f917a76cc203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 09:34:38.341631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:38.341031 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 09:34:38.341631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:38.341099 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" podUID="80566eca-1f41-450c-968e-f917a76cc203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 09:34:38.757330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:38.757286 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95"] Oct 30 09:34:38.757624 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:38.757569 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" podUID="f2176b3a-82cf-41b4-ae69-c5dea2a721f7" containerName="istio-proxy" containerID="cri-o://297ec831868790705f1be4942c173e7cde72fd7981162e83b40d393d8581b068" gracePeriod=30 Oct 30 09:34:39.343152 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:39.343118 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:39.343665 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:39.343644 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:39.344624 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:39.344568 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:34:43.105693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:43.105664 5938 scope.go:122] "RemoveContainer" containerID="28e9e33ea7238b9b78196c0403903e5014b91792dc65e79c17b330e3ebf01007" Oct 30 09:34:43.118137 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:43.118116 5938 scope.go:122] "RemoveContainer" containerID="cd568b90c4efc1ced627c3429f39505a5626a96047250d7ce6d9e98b58bfab9f" Oct 30 09:34:43.128736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:43.128698 5938 scope.go:122] "RemoveContainer" containerID="6ae07e9659cf64e83e872578ae582f2f6589b748bad5ed6a13f2f06d686309fc" Oct 30 09:34:44.030294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.030209 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:34:44.068691 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.068651 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-podinfo\") pod \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " Oct 30 09:34:44.068901 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.068708 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-envoy\") pod \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " Oct 30 09:34:44.068901 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.068737 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-workload-socket\") pod \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " Oct 30 09:34:44.068901 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.068781 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-data\") pod \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " Oct 30 09:34:44.068901 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.068805 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-credential-socket\") pod \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " Oct 30 09:34:44.069113 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.068997 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-token\" (UniqueName: \"kubernetes.io/projected/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-token\") pod \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " Oct 30 09:34:44.069113 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.069043 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-kube-api-access-sjhzg\" (UniqueName: \"kubernetes.io/projected/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-kube-api-access-sjhzg\") pod \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " Oct 30 09:34:44.069218 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.069189 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-workload-certs\") pod \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " Oct 30 09:34:44.069312 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.069241 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-data" pod "f2176b3a-82cf-41b4-ae69-c5dea2a721f7" (UID: "f2176b3a-82cf-41b4-ae69-c5dea2a721f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:44.069375 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.069262 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istiod-ca-cert\") pod \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\" (UID: \"f2176b3a-82cf-41b4-ae69-c5dea2a721f7\") " Oct 30 09:34:44.069375 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.069341 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-credential-socket" pod "f2176b3a-82cf-41b4-ae69-c5dea2a721f7" (UID: "f2176b3a-82cf-41b4-ae69-c5dea2a721f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:44.069698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.069652 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-workload-certs" pod "f2176b3a-82cf-41b4-ae69-c5dea2a721f7" (UID: "f2176b3a-82cf-41b4-ae69-c5dea2a721f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:44.069698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.069668 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istiod-ca-cert" pod "f2176b3a-82cf-41b4-ae69-c5dea2a721f7" (UID: "f2176b3a-82cf-41b4-ae69-c5dea2a721f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:34:44.069868 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.069143 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-workload-socket" pod "f2176b3a-82cf-41b4-ae69-c5dea2a721f7" (UID: "f2176b3a-82cf-41b4-ae69-c5dea2a721f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:44.070142 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.070113 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:44.070142 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.070144 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:44.070344 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.070159 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:44.070344 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.070173 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:44.070344 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.070241 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:44.071539 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.071505 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-podinfo" pod "f2176b3a-82cf-41b4-ae69-c5dea2a721f7" (UID: "f2176b3a-82cf-41b4-ae69-c5dea2a721f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:34:44.071791 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.071761 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-token" pod "f2176b3a-82cf-41b4-ae69-c5dea2a721f7" (UID: "f2176b3a-82cf-41b4-ae69-c5dea2a721f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:34:44.071922 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.071828 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-envoy" pod "f2176b3a-82cf-41b4-ae69-c5dea2a721f7" (UID: "f2176b3a-82cf-41b4-ae69-c5dea2a721f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:44.071981 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.071940 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-kube-api-access-sjhzg" pod "f2176b3a-82cf-41b4-ae69-c5dea2a721f7" (UID: "f2176b3a-82cf-41b4-ae69-c5dea2a721f7"). InnerVolumeSpecName "kube-api-access-sjhzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:34:44.172028 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.171984 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:44.172028 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.172021 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:44.172028 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.172033 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:44.172505 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.172041 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjhzg\" (UniqueName: \"kubernetes.io/projected/f2176b3a-82cf-41b4-ae69-c5dea2a721f7-kube-api-access-sjhzg\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:34:44.293471 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.293376 5938 generic.go:356] "Generic (PLEG): container finished" podID="f2176b3a-82cf-41b4-ae69-c5dea2a721f7" containerID="297ec831868790705f1be4942c173e7cde72fd7981162e83b40d393d8581b068" exitCode=0 Oct 30 09:34:44.293471 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.293441 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" Oct 30 09:34:44.293471 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.293434 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" event={"ID":"f2176b3a-82cf-41b4-ae69-c5dea2a721f7","Type":"ContainerDied","Data":"297ec831868790705f1be4942c173e7cde72fd7981162e83b40d393d8581b068"} Oct 30 09:34:44.293761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.293506 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95" event={"ID":"f2176b3a-82cf-41b4-ae69-c5dea2a721f7","Type":"ContainerDied","Data":"a7e2c63429b03d01c1e0ab9b2ebdb8ea466584458487906a31d38cb2c466cbdc"} Oct 30 09:34:44.293761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.293537 5938 scope.go:122] "RemoveContainer" containerID="297ec831868790705f1be4942c173e7cde72fd7981162e83b40d393d8581b068" Oct 30 09:34:44.304535 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.304508 5938 scope.go:122] "RemoveContainer" containerID="297ec831868790705f1be4942c173e7cde72fd7981162e83b40d393d8581b068" Oct 30 09:34:44.304948 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:34:44.304920 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"297ec831868790705f1be4942c173e7cde72fd7981162e83b40d393d8581b068\": container with ID starting with 297ec831868790705f1be4942c173e7cde72fd7981162e83b40d393d8581b068 not found: ID does not exist" containerID="297ec831868790705f1be4942c173e7cde72fd7981162e83b40d393d8581b068" Oct 30 09:34:44.305010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.304960 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"297ec831868790705f1be4942c173e7cde72fd7981162e83b40d393d8581b068"} err="failed to get container status \"297ec831868790705f1be4942c173e7cde72fd7981162e83b40d393d8581b068\": rpc error: code = NotFound desc = could not find container \"297ec831868790705f1be4942c173e7cde72fd7981162e83b40d393d8581b068\": container with ID starting with 297ec831868790705f1be4942c173e7cde72fd7981162e83b40d393d8581b068 not found: ID does not exist" Oct 30 09:34:44.351885 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.351849 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95"] Oct 30 09:34:44.355371 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:44.355338 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p6d95"] Oct 30 09:34:45.035941 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:45.035905 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh"] Oct 30 09:34:45.041120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:45.041088 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41a196ec-dgkqh"] Oct 30 09:34:45.811839 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:34:45.811786 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:34:46.257917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:46.257882 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef64d976-b243-4850-9a85-de7e9483bdbf" path="/var/lib/kubelet/pods/ef64d976-b243-4850-9a85-de7e9483bdbf/volumes" Oct 30 09:34:46.258419 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:46.258402 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2176b3a-82cf-41b4-ae69-c5dea2a721f7" path="/var/lib/kubelet/pods/f2176b3a-82cf-41b4-ae69-c5dea2a721f7/volumes" Oct 30 09:34:50.029161 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:50.029122 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92"] Oct 30 09:34:50.042155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:50.042113 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4cf242f-4dw92"] Oct 30 09:34:50.261018 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:50.260979 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c4fbd3-24cb-46f6-ad01-962114ae7ece" path="/var/lib/kubelet/pods/00c4fbd3-24cb-46f6-ad01-962114ae7ece/volumes" Oct 30 09:34:54.085605 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.085555 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw"] Oct 30 09:34:54.086229 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.086211 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f2176b3a-82cf-41b4-ae69-c5dea2a721f7" containerName="istio-proxy" Oct 30 09:34:54.086229 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.086230 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2176b3a-82cf-41b4-ae69-c5dea2a721f7" containerName="istio-proxy" Oct 30 09:34:54.086443 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.086430 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2176b3a-82cf-41b4-ae69-c5dea2a721f7" containerName="istio-proxy" Oct 30 09:34:54.092155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.092125 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.111510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.111476 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw"] Oct 30 09:34:54.121642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.120128 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-0dc148\"/\"kube-root-ca.crt\"" Oct 30 09:34:54.121642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.120464 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-0dc148\"/\"auth-test-model-0dc148-kserve-self-signed-certs\"" Oct 30 09:34:54.123707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.123604 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-0dc148\"/\"openshift-service-ca.crt\"" Oct 30 09:34:54.123891 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.123723 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-0dc148\"/\"default-dockercfg-6d9jj\"" Oct 30 09:34:54.239236 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.239192 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-dshm\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.239467 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.239356 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-model-cache\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.239467 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.239457 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qbfv\" (UniqueName: \"kubernetes.io/projected/4e15c89b-93ef-4515-a947-930e43925d64-kube-api-access-8qbfv\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.239614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.239488 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e15c89b-93ef-4515-a947-930e43925d64-tls-certs\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.239614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.239511 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-tmp-dir\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.239798 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.239771 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-home\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.346667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.346133 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-model-cache\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.346667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.346263 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qbfv\" (UniqueName: \"kubernetes.io/projected/4e15c89b-93ef-4515-a947-930e43925d64-kube-api-access-8qbfv\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.346667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.346296 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e15c89b-93ef-4515-a947-930e43925d64-tls-certs\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.346667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.346323 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-tmp-dir\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.346667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.346446 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-home\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.346667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.346610 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-dshm\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.357858 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.347790 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-model-cache\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.357858 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.348086 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-tmp-dir\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.357858 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.348357 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-home\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.361349 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.361315 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e15c89b-93ef-4515-a947-930e43925d64-tls-certs\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.369070 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.369036 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-dshm\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.425614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.411846 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qbfv\" (UniqueName: \"kubernetes.io/projected/4e15c89b-93ef-4515-a947-930e43925d64-kube-api-access-8qbfv\") pod \"auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.425614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.414945 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:54.780690 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:34:54.780646 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e15c89b_93ef_4515_a947_930e43925d64.slice/crio-c349f7b24be69e312e768d53128d1285ec5affcfe2bd35c89910ac9efff02cc9 WatchSource:0}: Error finding container c349f7b24be69e312e768d53128d1285ec5affcfe2bd35c89910ac9efff02cc9: Status 404 returned error can't find the container with id c349f7b24be69e312e768d53128d1285ec5affcfe2bd35c89910ac9efff02cc9 Oct 30 09:34:54.782209 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:54.782183 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw"] Oct 30 09:34:55.445850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:55.445806 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" event={"ID":"4e15c89b-93ef-4515-a947-930e43925d64","Type":"ContainerStarted","Data":"cd87fd74f6ab2155584bf6914e599a0667a64874c41cd75a3b98ce3a4868ea3e"} Oct 30 09:34:55.446390 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:55.445860 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" event={"ID":"4e15c89b-93ef-4515-a947-930e43925d64","Type":"ContainerStarted","Data":"c349f7b24be69e312e768d53128d1285ec5affcfe2bd35c89910ac9efff02cc9"} Oct 30 09:34:55.490018 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:55.489956 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" podStartSLOduration=1.489939325 podStartE2EDuration="1.489939325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:34:55.487081435 +0000 UTC m=+2413.349787257" watchObservedRunningTime="2026-10-30 09:34:55.489939325 +0000 UTC m=+2413.352645146" Oct 30 09:34:56.415552 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:56.415518 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:34:57.387625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.387566 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns"] Oct 30 09:34:57.393731 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.393698 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.412881 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.412849 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns"] Oct 30 09:34:57.421177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.421147 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-304459\"/\"openshift-service-ca.crt\"" Oct 30 09:34:57.421177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.421181 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-304459\"/\"kube-root-ca.crt\"" Oct 30 09:34:57.421414 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.421261 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-304459\"/\"default-dockercfg-rf6gf\"" Oct 30 09:34:57.426954 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.426892 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-304459\"/\"auth-test-model-304459-kserve-self-signed-certs\"" Oct 30 09:34:57.503878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.503838 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cd25eb1-3de8-48ff-92ae-06d486f584b6-tls-certs\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.504094 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.504066 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-model-cache\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.504145 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.504111 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-dshm\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.504265 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.504236 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-tmp-dir\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.504426 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.504402 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-home\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.504523 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.504475 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zckb7\" (UniqueName: \"kubernetes.io/projected/4cd25eb1-3de8-48ff-92ae-06d486f584b6-kube-api-access-zckb7\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.606229 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.606181 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-model-cache\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.606229 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.606231 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-dshm\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.607271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.606284 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-tmp-dir\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.607271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.606461 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-home\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.607271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.606528 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zckb7\" (UniqueName: \"kubernetes.io/projected/4cd25eb1-3de8-48ff-92ae-06d486f584b6-kube-api-access-zckb7\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.607271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.606690 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-model-cache\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.607271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.606703 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-tmp-dir\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.607271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.606711 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cd25eb1-3de8-48ff-92ae-06d486f584b6-tls-certs\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.607862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.607370 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-home\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.610673 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.609319 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-dshm\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.610673 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.609993 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cd25eb1-3de8-48ff-92ae-06d486f584b6-tls-certs\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.643672 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.637964 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zckb7\" (UniqueName: \"kubernetes.io/projected/4cd25eb1-3de8-48ff-92ae-06d486f584b6-kube-api-access-zckb7\") pod \"auth-test-model-304459-kserve-7cb68f67c-wz9ns\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:57.719613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:57.718795 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:58.052328 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:58.052285 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns"] Oct 30 09:34:58.052536 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:34:58.052512 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cd25eb1_3de8_48ff_92ae_06d486f584b6.slice/crio-516a32d1fa92c946066108d36d765d47950c06fc106d2fb1c4caa7e60dd50435 WatchSource:0}: Error finding container 516a32d1fa92c946066108d36d765d47950c06fc106d2fb1c4caa7e60dd50435: Status 404 returned error can't find the container with id 516a32d1fa92c946066108d36d765d47950c06fc106d2fb1c4caa7e60dd50435 Oct 30 09:34:58.474762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:58.474716 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" event={"ID":"4cd25eb1-3de8-48ff-92ae-06d486f584b6","Type":"ContainerStarted","Data":"ccd895375d0a741aea70b2a5fe18615ac0e9ff6a558bcae7077e6bf463c84b9a"} Oct 30 09:34:58.475268 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:58.474791 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" event={"ID":"4cd25eb1-3de8-48ff-92ae-06d486f584b6","Type":"ContainerStarted","Data":"516a32d1fa92c946066108d36d765d47950c06fc106d2fb1c4caa7e60dd50435"} Oct 30 09:34:58.523938 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:58.523851 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" podStartSLOduration=1.52382895 podStartE2EDuration="1.52382895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:34:58.516823791 +0000 UTC m=+2416.379529619" watchObservedRunningTime="2026-10-30 09:34:58.52382895 +0000 UTC m=+2416.386534754" Oct 30 09:34:58.719318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:58.719280 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:34:59.030011 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:59.029973 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq"] Oct 30 09:34:59.043042 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:34:59.043002 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bf585ce-vf8zq"] Oct 30 09:35:00.259778 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:00.259729 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653b44ab-fdf8-4f1a-bd3c-946173e53afe" path="/var/lib/kubelet/pods/653b44ab-fdf8-4f1a-bd3c-946173e53afe/volumes" Oct 30 09:35:04.416339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:04.416302 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:35:04.429497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:04.429465 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:35:04.534539 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:04.534493 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:35:07.719494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:07.719460 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:35:07.732727 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:07.732697 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:35:08.566904 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:08.566874 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:35:15.811553 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:35:15.811515 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:35:23.313660 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:23.313560 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x"] Oct 30 09:35:23.320464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:23.320434 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" Oct 30 09:35:23.337065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:23.337033 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:35:23.337375 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:23.337356 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:35:23.337818 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:23.337747 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x"] Oct 30 09:35:23.455237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:23.453955 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322acd4d-7be8-4e60-a336-5df474eaa116-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x\" (UID: \"322acd4d-7be8-4e60-a336-5df474eaa116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" Oct 30 09:35:23.455237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:23.454004 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322acd4d-7be8-4e60-a336-5df474eaa116-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x\" (UID: \"322acd4d-7be8-4e60-a336-5df474eaa116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" Oct 30 09:35:23.455237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:23.454334 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9hrq\" (UniqueName: \"kubernetes.io/projected/322acd4d-7be8-4e60-a336-5df474eaa116-kube-api-access-p9hrq\") pod \"maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x\" (UID: \"322acd4d-7be8-4e60-a336-5df474eaa116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" Oct 30 09:35:23.555646 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:23.555600 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9hrq\" (UniqueName: \"kubernetes.io/projected/322acd4d-7be8-4e60-a336-5df474eaa116-kube-api-access-p9hrq\") pod \"maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x\" (UID: \"322acd4d-7be8-4e60-a336-5df474eaa116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" Oct 30 09:35:23.555646 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:23.555655 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322acd4d-7be8-4e60-a336-5df474eaa116-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x\" (UID: \"322acd4d-7be8-4e60-a336-5df474eaa116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" Oct 30 09:35:23.555943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:23.555674 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322acd4d-7be8-4e60-a336-5df474eaa116-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x\" (UID: \"322acd4d-7be8-4e60-a336-5df474eaa116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" Oct 30 09:35:23.556617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:23.556575 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322acd4d-7be8-4e60-a336-5df474eaa116-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x\" (UID: \"322acd4d-7be8-4e60-a336-5df474eaa116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" Oct 30 09:35:23.558781 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:23.558754 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322acd4d-7be8-4e60-a336-5df474eaa116-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x\" (UID: \"322acd4d-7be8-4e60-a336-5df474eaa116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" Oct 30 09:35:23.576673 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:23.576568 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9hrq\" (UniqueName: \"kubernetes.io/projected/322acd4d-7be8-4e60-a336-5df474eaa116-kube-api-access-p9hrq\") pod \"maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x\" (UID: \"322acd4d-7be8-4e60-a336-5df474eaa116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" Oct 30 09:35:23.643981 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:23.643934 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" Oct 30 09:35:23.957134 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:23.957100 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x"] Oct 30 09:35:23.958260 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:35:23.958234 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322acd4d_7be8_4e60_a336_5df474eaa116.slice/crio-0a847ceed0972b00a76b65af2612ed43cb274991b5e4c7c54736dc23f48c79da WatchSource:0}: Error finding container 0a847ceed0972b00a76b65af2612ed43cb274991b5e4c7c54736dc23f48c79da: Status 404 returned error can't find the container with id 0a847ceed0972b00a76b65af2612ed43cb274991b5e4c7c54736dc23f48c79da Oct 30 09:35:24.697637 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:24.697577 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" event={"ID":"322acd4d-7be8-4e60-a336-5df474eaa116","Type":"ContainerStarted","Data":"db8bdf98a6e5271e11d68850b6569bb34423990fcdd2cc8f5d99f20422be6274"} Oct 30 09:35:24.698113 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:24.697670 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" event={"ID":"322acd4d-7be8-4e60-a336-5df474eaa116","Type":"ContainerStarted","Data":"0a847ceed0972b00a76b65af2612ed43cb274991b5e4c7c54736dc23f48c79da"} Oct 30 09:35:24.741357 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:24.741297 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" podStartSLOduration=1.741281731 podStartE2EDuration="1.741281731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:35:24.73812533 +0000 UTC m=+2442.600831152" watchObservedRunningTime="2026-10-30 09:35:24.741281731 +0000 UTC m=+2442.603987598" Oct 30 09:35:25.710376 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:25.710341 5938 generic.go:356] "Generic (PLEG): container finished" podID="322acd4d-7be8-4e60-a336-5df474eaa116" containerID="db8bdf98a6e5271e11d68850b6569bb34423990fcdd2cc8f5d99f20422be6274" exitCode=0 Oct 30 09:35:25.710882 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:25.710423 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" event={"ID":"322acd4d-7be8-4e60-a336-5df474eaa116","Type":"ContainerDied","Data":"db8bdf98a6e5271e11d68850b6569bb34423990fcdd2cc8f5d99f20422be6274"} Oct 30 09:35:26.898095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:26.898064 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" Oct 30 09:35:26.912472 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:26.912441 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/322acd4d-7be8-4e60-a336-5df474eaa116-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322acd4d-7be8-4e60-a336-5df474eaa116-maas-api-xks-ca\") pod \"322acd4d-7be8-4e60-a336-5df474eaa116\" (UID: \"322acd4d-7be8-4e60-a336-5df474eaa116\") " Oct 30 09:35:26.912665 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:26.912525 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322acd4d-7be8-4e60-a336-5df474eaa116-kube-api-access-p9hrq\" (UniqueName: \"kubernetes.io/projected/322acd4d-7be8-4e60-a336-5df474eaa116-kube-api-access-p9hrq\") pod \"322acd4d-7be8-4e60-a336-5df474eaa116\" (UID: \"322acd4d-7be8-4e60-a336-5df474eaa116\") " Oct 30 09:35:26.912665 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:26.912622 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322acd4d-7be8-4e60-a336-5df474eaa116-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322acd4d-7be8-4e60-a336-5df474eaa116-maas-api-openshift-ca\") pod \"322acd4d-7be8-4e60-a336-5df474eaa116\" (UID: \"322acd4d-7be8-4e60-a336-5df474eaa116\") " Oct 30 09:35:26.913036 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:26.913006 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322acd4d-7be8-4e60-a336-5df474eaa116-maas-api-openshift-ca" pod "322acd4d-7be8-4e60-a336-5df474eaa116" (UID: "322acd4d-7be8-4e60-a336-5df474eaa116"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:35:26.913160 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:26.913119 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322acd4d-7be8-4e60-a336-5df474eaa116-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:35:26.914865 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:26.914840 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322acd4d-7be8-4e60-a336-5df474eaa116-maas-api-xks-ca" pod "322acd4d-7be8-4e60-a336-5df474eaa116" (UID: "322acd4d-7be8-4e60-a336-5df474eaa116"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:35:26.915197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:26.915174 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322acd4d-7be8-4e60-a336-5df474eaa116-kube-api-access-p9hrq" pod "322acd4d-7be8-4e60-a336-5df474eaa116" (UID: "322acd4d-7be8-4e60-a336-5df474eaa116"). InnerVolumeSpecName "kube-api-access-p9hrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:35:27.014393 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:27.014301 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9hrq\" (UniqueName: \"kubernetes.io/projected/322acd4d-7be8-4e60-a336-5df474eaa116-kube-api-access-p9hrq\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:35:27.014393 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:27.014333 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322acd4d-7be8-4e60-a336-5df474eaa116-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:35:27.728565 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:27.728539 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" Oct 30 09:35:27.728773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:27.728574 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x" event={"ID":"322acd4d-7be8-4e60-a336-5df474eaa116","Type":"ContainerDied","Data":"0a847ceed0972b00a76b65af2612ed43cb274991b5e4c7c54736dc23f48c79da"} Oct 30 09:35:27.728773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:27.728646 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a847ceed0972b00a76b65af2612ed43cb274991b5e4c7c54736dc23f48c79da" Oct 30 09:35:35.047506 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:35.047473 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm"] Oct 30 09:35:35.062499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:35.062460 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-561cd2e4-kxptm"] Oct 30 09:35:36.257938 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:36.257900 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0031e67c-37b6-4b1d-ba17-a560d4aafe47" path="/var/lib/kubelet/pods/0031e67c-37b6-4b1d-ba17-a560d4aafe47/volumes" Oct 30 09:35:40.267031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:40.266979 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x"] Oct 30 09:35:40.277227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:40.277192 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-1413c8-bzc8x"] Oct 30 09:35:42.259608 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:42.259549 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="322acd4d-7be8-4e60-a336-5df474eaa116" path="/var/lib/kubelet/pods/322acd4d-7be8-4e60-a336-5df474eaa116/volumes" Oct 30 09:35:43.176811 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:43.176779 5938 scope.go:122] "RemoveContainer" containerID="d42ef0e0251601c89a8993c4a631c0d4c4dbbb0bab40322a679dc0d3b75fa0e3" Oct 30 09:35:43.189026 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:43.188996 5938 scope.go:122] "RemoveContainer" containerID="89289d2ec29d267cac8f4b3bebbf52ef977b8010d36da7f0b28aa2007dff86da" Oct 30 09:35:43.199563 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:43.199532 5938 scope.go:122] "RemoveContainer" containerID="d510acf079f385b28bff3b3c93031c9688f363a80a5d4264fe4ee31f7d390b63" Oct 30 09:35:43.210024 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:43.209997 5938 scope.go:122] "RemoveContainer" containerID="40eb453bfda17f20725b8f43cdb8e96b6e2dcee9bdb8a2054d70d8218769e9c2" Oct 30 09:35:43.220531 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:43.220506 5938 scope.go:122] "RemoveContainer" containerID="2c9ee1217a23799611dbaf0aed2a01b3064fd3c5cd1e343a1deea890fb82198f" Oct 30 09:35:45.812701 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:35:45.812665 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:35:51.057769 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:51.057718 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh"] Oct 30 09:35:51.071488 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:51.071453 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb699118-vjbrh"] Oct 30 09:35:52.261623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:35:52.261565 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07951f59-63ef-491b-9bb7-24038d2c5c5c" path="/var/lib/kubelet/pods/07951f59-63ef-491b-9bb7-24038d2c5c5c/volumes" Oct 30 09:36:04.038532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:04.038494 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t"] Oct 30 09:36:04.051681 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:04.051644 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bf3c8b6-btv9t"] Oct 30 09:36:04.267642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:04.267562 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ab290b-5408-493b-ba9c-0c3a651ec49b" path="/var/lib/kubelet/pods/d2ab290b-5408-493b-ba9c-0c3a651ec49b/volumes" Oct 30 09:36:15.811332 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:36:15.811289 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:36:16.030003 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:16.029958 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp"] Oct 30 09:36:16.042325 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:16.042285 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b218ff4-xb2lp"] Oct 30 09:36:16.263626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:16.263565 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09bafb89-2b8a-42bc-a0a4-8ba4eed62526" path="/var/lib/kubelet/pods/09bafb89-2b8a-42bc-a0a4-8ba4eed62526/volumes" Oct 30 09:36:23.030033 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:23.029993 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7"] Oct 30 09:36:23.042077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:23.042048 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a5c890-2gvq7"] Oct 30 09:36:24.258461 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:24.258421 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f41f71a6-94a3-4f27-8ce0-cba6940bfba5" path="/var/lib/kubelet/pods/f41f71a6-94a3-4f27-8ce0-cba6940bfba5/volumes" Oct 30 09:36:34.029175 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:34.029136 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm"] Oct 30 09:36:34.042282 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:34.042247 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f2560e-24mrm"] Oct 30 09:36:34.259367 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:34.259324 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79dc7829-7dcb-4118-a759-ed05d381e749" path="/var/lib/kubelet/pods/79dc7829-7dcb-4118-a759-ed05d381e749/volumes" Oct 30 09:36:36.026679 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:36.026641 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g"] Oct 30 09:36:36.040073 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:36.040042 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d3c9f02-tvf7g"] Oct 30 09:36:36.258655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:36.258618 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adf218ba-ee3e-44ba-a0e1-676804744021" path="/var/lib/kubelet/pods/adf218ba-ee3e-44ba-a0e1-676804744021/volumes" Oct 30 09:36:43.284733 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:43.284702 5938 scope.go:122] "RemoveContainer" containerID="c2152deb65a560a517e9910e623c7263bf6adf05101cbcb2ad2071dc1ab49efc" Oct 30 09:36:43.295706 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:43.295681 5938 scope.go:122] "RemoveContainer" containerID="22852ba1efc96d9bd4c280eeca1ba1f16f83d54c242eb5c6f8b2be3cf2dcf130" Oct 30 09:36:43.309089 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:43.309065 5938 scope.go:122] "RemoveContainer" containerID="3214b73aa64b7227c1e342d43e6116701bdbc1bc83f475c4b6a73dc6fc502e66" Oct 30 09:36:45.028407 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:45.028368 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v"] Oct 30 09:36:45.041637 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:45.041571 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264868e2-99f8v"] Oct 30 09:36:45.810906 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:36:45.810866 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:36:46.258165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:46.258130 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab08dc6e-1b67-4d88-83d6-24edf1376127" path="/var/lib/kubelet/pods/ab08dc6e-1b67-4d88-83d6-24edf1376127/volumes" Oct 30 09:36:50.479691 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.479659 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs"] Oct 30 09:36:50.480346 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.480328 5938 cpu_manager.go:414] "containerMap: removing container" podUID="322acd4d-7be8-4e60-a336-5df474eaa116" containerName="revoke-keys" Oct 30 09:36:50.480421 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.480346 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322acd4d-7be8-4e60-a336-5df474eaa116" containerName="revoke-keys" Oct 30 09:36:50.480493 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.480483 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="322acd4d-7be8-4e60-a336-5df474eaa116" containerName="revoke-keys" Oct 30 09:36:50.485648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.485623 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs" Oct 30 09:36:50.502149 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.502123 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs"] Oct 30 09:36:50.515005 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.514970 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:36:50.515257 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.515211 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:36:50.609542 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.609505 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wpkw\" (UniqueName: \"kubernetes.io/projected/87e254ef-f6d0-47df-9424-0f7210062ee9-kube-api-access-8wpkw\") pod \"maas-api-revoke-sub-214709f1-gn4xs\" (UID: \"87e254ef-f6d0-47df-9424-0f7210062ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs" Oct 30 09:36:50.609875 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.609851 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e254ef-f6d0-47df-9424-0f7210062ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-214709f1-gn4xs\" (UID: \"87e254ef-f6d0-47df-9424-0f7210062ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs" Oct 30 09:36:50.610037 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.609972 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e254ef-f6d0-47df-9424-0f7210062ee9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-214709f1-gn4xs\" (UID: \"87e254ef-f6d0-47df-9424-0f7210062ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs" Oct 30 09:36:50.711745 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.711709 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wpkw\" (UniqueName: \"kubernetes.io/projected/87e254ef-f6d0-47df-9424-0f7210062ee9-kube-api-access-8wpkw\") pod \"maas-api-revoke-sub-214709f1-gn4xs\" (UID: \"87e254ef-f6d0-47df-9424-0f7210062ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs" Oct 30 09:36:50.711925 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.711901 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e254ef-f6d0-47df-9424-0f7210062ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-214709f1-gn4xs\" (UID: \"87e254ef-f6d0-47df-9424-0f7210062ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs" Oct 30 09:36:50.711976 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.711944 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e254ef-f6d0-47df-9424-0f7210062ee9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-214709f1-gn4xs\" (UID: \"87e254ef-f6d0-47df-9424-0f7210062ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs" Oct 30 09:36:50.712574 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.712549 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e254ef-f6d0-47df-9424-0f7210062ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-214709f1-gn4xs\" (UID: \"87e254ef-f6d0-47df-9424-0f7210062ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs" Oct 30 09:36:50.714530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.714509 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e254ef-f6d0-47df-9424-0f7210062ee9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-214709f1-gn4xs\" (UID: \"87e254ef-f6d0-47df-9424-0f7210062ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs" Oct 30 09:36:50.733418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.733346 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wpkw\" (UniqueName: \"kubernetes.io/projected/87e254ef-f6d0-47df-9424-0f7210062ee9-kube-api-access-8wpkw\") pod \"maas-api-revoke-sub-214709f1-gn4xs\" (UID: \"87e254ef-f6d0-47df-9424-0f7210062ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs" Oct 30 09:36:50.800484 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:50.800445 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs" Oct 30 09:36:51.027442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:51.027345 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7"] Oct 30 09:36:51.030127 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:51.030102 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb21f57c-jccz7"] Oct 30 09:36:51.107470 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:51.107446 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs"] Oct 30 09:36:51.109624 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:36:51.109581 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e254ef_f6d0_47df_9424_0f7210062ee9.slice/crio-a775bda94cf54ed00dd1bc26a8fa3a24ec78219a95c85be0d65e26aebe8d719d WatchSource:0}: Error finding container a775bda94cf54ed00dd1bc26a8fa3a24ec78219a95c85be0d65e26aebe8d719d: Status 404 returned error can't find the container with id a775bda94cf54ed00dd1bc26a8fa3a24ec78219a95c85be0d65e26aebe8d719d Oct 30 09:36:51.428750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:51.428711 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs" event={"ID":"87e254ef-f6d0-47df-9424-0f7210062ee9","Type":"ContainerStarted","Data":"a775bda94cf54ed00dd1bc26a8fa3a24ec78219a95c85be0d65e26aebe8d719d"} Oct 30 09:36:52.260025 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:52.259994 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8931547c-3aea-456f-8eef-27f96da59180" path="/var/lib/kubelet/pods/8931547c-3aea-456f-8eef-27f96da59180/volumes" Oct 30 09:36:52.437809 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:52.437774 5938 generic.go:356] "Generic (PLEG): container finished" podID="87e254ef-f6d0-47df-9424-0f7210062ee9" containerID="bc3a6d96874b488ae9ad907f7a93ac7266ac12bc00f9270f84ff87a8cb5a5131" exitCode=0 Oct 30 09:36:52.438046 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:52.437860 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs" event={"ID":"87e254ef-f6d0-47df-9424-0f7210062ee9","Type":"ContainerDied","Data":"bc3a6d96874b488ae9ad907f7a93ac7266ac12bc00f9270f84ff87a8cb5a5131"} Oct 30 09:36:53.633710 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:53.633684 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs" Oct 30 09:36:53.655850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:53.655820 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87e254ef-f6d0-47df-9424-0f7210062ee9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e254ef-f6d0-47df-9424-0f7210062ee9-maas-api-xks-ca\") pod \"87e254ef-f6d0-47df-9424-0f7210062ee9\" (UID: \"87e254ef-f6d0-47df-9424-0f7210062ee9\") " Oct 30 09:36:53.656029 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:53.655916 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e254ef-f6d0-47df-9424-0f7210062ee9-kube-api-access-8wpkw\" (UniqueName: \"kubernetes.io/projected/87e254ef-f6d0-47df-9424-0f7210062ee9-kube-api-access-8wpkw\") pod \"87e254ef-f6d0-47df-9424-0f7210062ee9\" (UID: \"87e254ef-f6d0-47df-9424-0f7210062ee9\") " Oct 30 09:36:53.656029 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:53.655941 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e254ef-f6d0-47df-9424-0f7210062ee9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e254ef-f6d0-47df-9424-0f7210062ee9-maas-api-openshift-ca\") pod \"87e254ef-f6d0-47df-9424-0f7210062ee9\" (UID: \"87e254ef-f6d0-47df-9424-0f7210062ee9\") " Oct 30 09:36:53.656360 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:53.656336 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e254ef-f6d0-47df-9424-0f7210062ee9-maas-api-openshift-ca" pod "87e254ef-f6d0-47df-9424-0f7210062ee9" (UID: "87e254ef-f6d0-47df-9424-0f7210062ee9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:36:53.658121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:53.658094 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87e254ef-f6d0-47df-9424-0f7210062ee9-maas-api-xks-ca" pod "87e254ef-f6d0-47df-9424-0f7210062ee9" (UID: "87e254ef-f6d0-47df-9424-0f7210062ee9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:36:53.658213 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:53.658155 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e254ef-f6d0-47df-9424-0f7210062ee9-kube-api-access-8wpkw" pod "87e254ef-f6d0-47df-9424-0f7210062ee9" (UID: "87e254ef-f6d0-47df-9424-0f7210062ee9"). InnerVolumeSpecName "kube-api-access-8wpkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:36:53.757499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:53.757413 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e254ef-f6d0-47df-9424-0f7210062ee9-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:36:53.757499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:53.757443 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wpkw\" (UniqueName: \"kubernetes.io/projected/87e254ef-f6d0-47df-9424-0f7210062ee9-kube-api-access-8wpkw\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:36:53.757499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:53.757453 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e254ef-f6d0-47df-9424-0f7210062ee9-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:36:54.455440 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:54.455411 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs" Oct 30 09:36:54.455440 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:54.455431 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs" event={"ID":"87e254ef-f6d0-47df-9424-0f7210062ee9","Type":"ContainerDied","Data":"a775bda94cf54ed00dd1bc26a8fa3a24ec78219a95c85be0d65e26aebe8d719d"} Oct 30 09:36:54.455677 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:54.455497 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a775bda94cf54ed00dd1bc26a8fa3a24ec78219a95c85be0d65e26aebe8d719d" Oct 30 09:36:59.028303 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:59.028269 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r"] Oct 30 09:36:59.042179 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:36:59.042146 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e5099a-hh84r"] Oct 30 09:37:00.258278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.258246 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1227bf8a-0716-48eb-9530-86ae66f12acd" path="/var/lib/kubelet/pods/1227bf8a-0716-48eb-9530-86ae66f12acd/volumes" Oct 30 09:37:00.803747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.803710 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4"] Oct 30 09:37:00.804322 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.804308 5938 cpu_manager.go:414] "containerMap: removing container" podUID="87e254ef-f6d0-47df-9424-0f7210062ee9" containerName="revoke-keys" Oct 30 09:37:00.804384 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.804325 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e254ef-f6d0-47df-9424-0f7210062ee9" containerName="revoke-keys" Oct 30 09:37:00.804457 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.804448 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e254ef-f6d0-47df-9424-0f7210062ee9" containerName="revoke-keys" Oct 30 09:37:00.808330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.808313 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4" Oct 30 09:37:00.827127 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.827089 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4"] Oct 30 09:37:00.836856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.836802 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:37:00.837034 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.836881 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:37:00.864980 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.864949 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wrwb\" (UniqueName: \"kubernetes.io/projected/a61b1d50-c810-4bcb-a406-f884a8f758f9-kube-api-access-5wrwb\") pod \"maas-api-revoke-sub-0acde49c-xxgc4\" (UID: \"a61b1d50-c810-4bcb-a406-f884a8f758f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4" Oct 30 09:37:00.865224 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.865205 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61b1d50-c810-4bcb-a406-f884a8f758f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0acde49c-xxgc4\" (UID: \"a61b1d50-c810-4bcb-a406-f884a8f758f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4" Oct 30 09:37:00.865366 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.865345 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61b1d50-c810-4bcb-a406-f884a8f758f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0acde49c-xxgc4\" (UID: \"a61b1d50-c810-4bcb-a406-f884a8f758f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4" Oct 30 09:37:00.967278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.967231 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61b1d50-c810-4bcb-a406-f884a8f758f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0acde49c-xxgc4\" (UID: \"a61b1d50-c810-4bcb-a406-f884a8f758f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4" Oct 30 09:37:00.967476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.967330 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61b1d50-c810-4bcb-a406-f884a8f758f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0acde49c-xxgc4\" (UID: \"a61b1d50-c810-4bcb-a406-f884a8f758f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4" Oct 30 09:37:00.967476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.967412 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wrwb\" (UniqueName: \"kubernetes.io/projected/a61b1d50-c810-4bcb-a406-f884a8f758f9-kube-api-access-5wrwb\") pod \"maas-api-revoke-sub-0acde49c-xxgc4\" (UID: \"a61b1d50-c810-4bcb-a406-f884a8f758f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4" Oct 30 09:37:00.968127 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.968100 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61b1d50-c810-4bcb-a406-f884a8f758f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0acde49c-xxgc4\" (UID: \"a61b1d50-c810-4bcb-a406-f884a8f758f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4" Oct 30 09:37:00.969828 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.969804 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61b1d50-c810-4bcb-a406-f884a8f758f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0acde49c-xxgc4\" (UID: \"a61b1d50-c810-4bcb-a406-f884a8f758f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4" Oct 30 09:37:00.988492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:00.988466 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wrwb\" (UniqueName: \"kubernetes.io/projected/a61b1d50-c810-4bcb-a406-f884a8f758f9-kube-api-access-5wrwb\") pod \"maas-api-revoke-sub-0acde49c-xxgc4\" (UID: \"a61b1d50-c810-4bcb-a406-f884a8f758f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4" Oct 30 09:37:01.122549 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:01.122516 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4" Oct 30 09:37:01.426625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:01.426580 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4"] Oct 30 09:37:01.428091 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:37:01.428067 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61b1d50_c810_4bcb_a406_f884a8f758f9.slice/crio-14100ac23af68267886a7f84a9f513680210f7b94eb557105503460f6444269e WatchSource:0}: Error finding container 14100ac23af68267886a7f84a9f513680210f7b94eb557105503460f6444269e: Status 404 returned error can't find the container with id 14100ac23af68267886a7f84a9f513680210f7b94eb557105503460f6444269e Oct 30 09:37:01.507458 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:01.507427 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4" event={"ID":"a61b1d50-c810-4bcb-a406-f884a8f758f9","Type":"ContainerStarted","Data":"14100ac23af68267886a7f84a9f513680210f7b94eb557105503460f6444269e"} Oct 30 09:37:02.516794 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:02.516762 5938 generic.go:356] "Generic (PLEG): container finished" podID="a61b1d50-c810-4bcb-a406-f884a8f758f9" containerID="5f65cfa20445e1800f97b151e42be24f5ca5ad066a8f98c5608169b3f8270c48" exitCode=0 Oct 30 09:37:02.517209 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:02.516803 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4" event={"ID":"a61b1d50-c810-4bcb-a406-f884a8f758f9","Type":"ContainerDied","Data":"5f65cfa20445e1800f97b151e42be24f5ca5ad066a8f98c5608169b3f8270c48"} Oct 30 09:37:03.695127 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:03.695018 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4" Oct 30 09:37:03.810267 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:03.810172 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a61b1d50-c810-4bcb-a406-f884a8f758f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61b1d50-c810-4bcb-a406-f884a8f758f9-maas-api-xks-ca\") pod \"a61b1d50-c810-4bcb-a406-f884a8f758f9\" (UID: \"a61b1d50-c810-4bcb-a406-f884a8f758f9\") " Oct 30 09:37:03.810267 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:03.810262 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a61b1d50-c810-4bcb-a406-f884a8f758f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61b1d50-c810-4bcb-a406-f884a8f758f9-maas-api-openshift-ca\") pod \"a61b1d50-c810-4bcb-a406-f884a8f758f9\" (UID: \"a61b1d50-c810-4bcb-a406-f884a8f758f9\") " Oct 30 09:37:03.810573 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:03.810323 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a61b1d50-c810-4bcb-a406-f884a8f758f9-kube-api-access-5wrwb\" (UniqueName: \"kubernetes.io/projected/a61b1d50-c810-4bcb-a406-f884a8f758f9-kube-api-access-5wrwb\") pod \"a61b1d50-c810-4bcb-a406-f884a8f758f9\" (UID: \"a61b1d50-c810-4bcb-a406-f884a8f758f9\") " Oct 30 09:37:03.810700 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:03.810672 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61b1d50-c810-4bcb-a406-f884a8f758f9-maas-api-openshift-ca" pod "a61b1d50-c810-4bcb-a406-f884a8f758f9" (UID: "a61b1d50-c810-4bcb-a406-f884a8f758f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:37:03.810807 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:03.810793 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61b1d50-c810-4bcb-a406-f884a8f758f9-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:03.812674 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:03.812646 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61b1d50-c810-4bcb-a406-f884a8f758f9-kube-api-access-5wrwb" pod "a61b1d50-c810-4bcb-a406-f884a8f758f9" (UID: "a61b1d50-c810-4bcb-a406-f884a8f758f9"). InnerVolumeSpecName "kube-api-access-5wrwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:03.812674 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:03.812658 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a61b1d50-c810-4bcb-a406-f884a8f758f9-maas-api-xks-ca" pod "a61b1d50-c810-4bcb-a406-f884a8f758f9" (UID: "a61b1d50-c810-4bcb-a406-f884a8f758f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:37:03.912431 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:03.912399 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61b1d50-c810-4bcb-a406-f884a8f758f9-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:03.912431 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:03.912428 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wrwb\" (UniqueName: \"kubernetes.io/projected/a61b1d50-c810-4bcb-a406-f884a8f758f9-kube-api-access-5wrwb\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:04.532913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:04.532835 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4" Oct 30 09:37:04.532913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:04.532844 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4" event={"ID":"a61b1d50-c810-4bcb-a406-f884a8f758f9","Type":"ContainerDied","Data":"14100ac23af68267886a7f84a9f513680210f7b94eb557105503460f6444269e"} Oct 30 09:37:04.532913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:04.532897 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14100ac23af68267886a7f84a9f513680210f7b94eb557105503460f6444269e" Oct 30 09:37:05.028483 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:05.028445 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf"] Oct 30 09:37:05.041959 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:05.041928 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-705a1a6c-pzrwf"] Oct 30 09:37:06.258153 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:06.258122 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f98c0a5-33ff-4e3b-946d-1b50bc80502b" path="/var/lib/kubelet/pods/4f98c0a5-33ff-4e3b-946d-1b50bc80502b/volumes" Oct 30 09:37:11.029269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:11.029237 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27"] Oct 30 09:37:11.042742 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:11.042715 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecbf83a1-fpc27"] Oct 30 09:37:12.258694 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:12.258657 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80bb7eaf-6ff2-4138-8a40-c5ed799a1123" path="/var/lib/kubelet/pods/80bb7eaf-6ff2-4138-8a40-c5ed799a1123/volumes" Oct 30 09:37:12.897846 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:12.897811 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c"] Oct 30 09:37:15.586628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:15.584482 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns"] Oct 30 09:37:15.824636 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:37:15.823317 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:37:27.923411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:27.923361 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" podUID="957b9998-b762-4673-bf29-1d5b8d7ae904" containerName="main" containerID="cri-o://0c0f5c22f720518e8f00462ed57f9bfedf665d73cdc62e506cb50b51ab7f3b9b" gracePeriod=45 Oct 30 09:37:28.360748 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.360718 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:37:28.501235 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.501187 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957b9998-b762-4673-bf29-1d5b8d7ae904-kube-api-access-w4r26\" (UniqueName: \"kubernetes.io/projected/957b9998-b762-4673-bf29-1d5b8d7ae904-kube-api-access-w4r26\") pod \"957b9998-b762-4673-bf29-1d5b8d7ae904\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " Oct 30 09:37:28.501454 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.501274 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-home\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-home\") pod \"957b9998-b762-4673-bf29-1d5b8d7ae904\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " Oct 30 09:37:28.501454 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.501317 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-model-cache\") pod \"957b9998-b762-4673-bf29-1d5b8d7ae904\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " Oct 30 09:37:28.501454 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.501382 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/957b9998-b762-4673-bf29-1d5b8d7ae904-tls-certs\" (UniqueName: \"kubernetes.io/secret/957b9998-b762-4673-bf29-1d5b8d7ae904-tls-certs\") pod \"957b9998-b762-4673-bf29-1d5b8d7ae904\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " Oct 30 09:37:28.501690 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.501542 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-tmp-dir\") pod \"957b9998-b762-4673-bf29-1d5b8d7ae904\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " Oct 30 09:37:28.501690 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.501546 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-home" pod "957b9998-b762-4673-bf29-1d5b8d7ae904" (UID: "957b9998-b762-4673-bf29-1d5b8d7ae904"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:28.501797 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.501746 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-model-cache" pod "957b9998-b762-4673-bf29-1d5b8d7ae904" (UID: "957b9998-b762-4673-bf29-1d5b8d7ae904"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:28.501797 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.501770 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-dshm\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-dshm\") pod \"957b9998-b762-4673-bf29-1d5b8d7ae904\" (UID: \"957b9998-b762-4673-bf29-1d5b8d7ae904\") " Oct 30 09:37:28.501910 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.501803 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-tmp-dir" pod "957b9998-b762-4673-bf29-1d5b8d7ae904" (UID: "957b9998-b762-4673-bf29-1d5b8d7ae904"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:28.502409 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.502384 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:28.502409 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.502409 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:28.502644 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.502420 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:28.503995 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.503975 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957b9998-b762-4673-bf29-1d5b8d7ae904-kube-api-access-w4r26" pod "957b9998-b762-4673-bf29-1d5b8d7ae904" (UID: "957b9998-b762-4673-bf29-1d5b8d7ae904"). InnerVolumeSpecName "kube-api-access-w4r26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:28.504354 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.504327 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957b9998-b762-4673-bf29-1d5b8d7ae904-tls-certs" pod "957b9998-b762-4673-bf29-1d5b8d7ae904" (UID: "957b9998-b762-4673-bf29-1d5b8d7ae904"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:37:28.504451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.504402 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-dshm" pod "957b9998-b762-4673-bf29-1d5b8d7ae904" (UID: "957b9998-b762-4673-bf29-1d5b8d7ae904"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:28.604197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.604122 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/957b9998-b762-4673-bf29-1d5b8d7ae904-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:28.604197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.604150 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/957b9998-b762-4673-bf29-1d5b8d7ae904-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:28.604197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.604160 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4r26\" (UniqueName: \"kubernetes.io/projected/957b9998-b762-4673-bf29-1d5b8d7ae904-kube-api-access-w4r26\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:28.715146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.715106 5938 generic.go:356] "Generic (PLEG): container finished" podID="957b9998-b762-4673-bf29-1d5b8d7ae904" containerID="0c0f5c22f720518e8f00462ed57f9bfedf665d73cdc62e506cb50b51ab7f3b9b" exitCode=0 Oct 30 09:37:28.715311 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.715183 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" Oct 30 09:37:28.715311 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.715187 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" event={"ID":"957b9998-b762-4673-bf29-1d5b8d7ae904","Type":"ContainerDied","Data":"0c0f5c22f720518e8f00462ed57f9bfedf665d73cdc62e506cb50b51ab7f3b9b"} Oct 30 09:37:28.715311 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.715300 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c" event={"ID":"957b9998-b762-4673-bf29-1d5b8d7ae904","Type":"ContainerDied","Data":"2b29a7b38f0dc73e113b5421ede3398e188e867ee52cb459e0afb34798840577"} Oct 30 09:37:28.715479 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.715318 5938 scope.go:122] "RemoveContainer" containerID="0c0f5c22f720518e8f00462ed57f9bfedf665d73cdc62e506cb50b51ab7f3b9b" Oct 30 09:37:28.731224 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.731198 5938 scope.go:122] "RemoveContainer" containerID="0c0f5c22f720518e8f00462ed57f9bfedf665d73cdc62e506cb50b51ab7f3b9b" Oct 30 09:37:28.731648 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:37:28.731622 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0f5c22f720518e8f00462ed57f9bfedf665d73cdc62e506cb50b51ab7f3b9b\": container with ID starting with 0c0f5c22f720518e8f00462ed57f9bfedf665d73cdc62e506cb50b51ab7f3b9b not found: ID does not exist" containerID="0c0f5c22f720518e8f00462ed57f9bfedf665d73cdc62e506cb50b51ab7f3b9b" Oct 30 09:37:28.731753 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.731658 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0f5c22f720518e8f00462ed57f9bfedf665d73cdc62e506cb50b51ab7f3b9b"} err="failed to get container status \"0c0f5c22f720518e8f00462ed57f9bfedf665d73cdc62e506cb50b51ab7f3b9b\": rpc error: code = NotFound desc = could not find container \"0c0f5c22f720518e8f00462ed57f9bfedf665d73cdc62e506cb50b51ab7f3b9b\": container with ID starting with 0c0f5c22f720518e8f00462ed57f9bfedf665d73cdc62e506cb50b51ab7f3b9b not found: ID does not exist" Oct 30 09:37:28.776340 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.776312 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c"] Oct 30 09:37:28.789487 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:28.789458 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-19c94a/auth-test-model-19c94a-kserve-744b56c457-qgl2c"] Oct 30 09:37:30.268325 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:30.268290 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="957b9998-b762-4673-bf29-1d5b8d7ae904" path="/var/lib/kubelet/pods/957b9998-b762-4673-bf29-1d5b8d7ae904/volumes" Oct 30 09:37:30.619862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:30.619796 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" podUID="822b48c6-2e86-45a3-af80-ef8cd9cf5674" containerName="main" containerID="cri-o://3ba907c858e6fa8cc1dea62b5656d6e9c65975a4872d8797b17efff5590da984" gracePeriod=45 Oct 30 09:37:30.940014 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:30.939983 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:37:31.052392 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.052363 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-model-cache\") pod \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " Oct 30 09:37:31.052551 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.052467 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-home\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-home\") pod \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " Oct 30 09:37:31.052551 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.052519 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-dshm\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-dshm\") pod \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " Oct 30 09:37:31.052685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.052572 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/822b48c6-2e86-45a3-af80-ef8cd9cf5674-tls-certs\" (UniqueName: \"kubernetes.io/secret/822b48c6-2e86-45a3-af80-ef8cd9cf5674-tls-certs\") pod \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " Oct 30 09:37:31.052685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.052649 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-tmp-dir\") pod \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " Oct 30 09:37:31.052791 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.052684 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-model-cache" pod "822b48c6-2e86-45a3-af80-ef8cd9cf5674" (UID: "822b48c6-2e86-45a3-af80-ef8cd9cf5674"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:31.052791 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.052687 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822b48c6-2e86-45a3-af80-ef8cd9cf5674-kube-api-access-q8546\" (UniqueName: \"kubernetes.io/projected/822b48c6-2e86-45a3-af80-ef8cd9cf5674-kube-api-access-q8546\") pod \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\" (UID: \"822b48c6-2e86-45a3-af80-ef8cd9cf5674\") " Oct 30 09:37:31.052791 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.052768 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-home" pod "822b48c6-2e86-45a3-af80-ef8cd9cf5674" (UID: "822b48c6-2e86-45a3-af80-ef8cd9cf5674"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:31.053053 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.052950 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-tmp-dir" pod "822b48c6-2e86-45a3-af80-ef8cd9cf5674" (UID: "822b48c6-2e86-45a3-af80-ef8cd9cf5674"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:31.053632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.053606 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:31.053632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.053631 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:31.053824 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.053640 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:31.055209 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.055182 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822b48c6-2e86-45a3-af80-ef8cd9cf5674-tls-certs" pod "822b48c6-2e86-45a3-af80-ef8cd9cf5674" (UID: "822b48c6-2e86-45a3-af80-ef8cd9cf5674"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:37:31.055309 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.055215 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-dshm" pod "822b48c6-2e86-45a3-af80-ef8cd9cf5674" (UID: "822b48c6-2e86-45a3-af80-ef8cd9cf5674"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:31.055352 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.055313 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822b48c6-2e86-45a3-af80-ef8cd9cf5674-kube-api-access-q8546" pod "822b48c6-2e86-45a3-af80-ef8cd9cf5674" (UID: "822b48c6-2e86-45a3-af80-ef8cd9cf5674"). InnerVolumeSpecName "kube-api-access-q8546". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:31.155324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.155288 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/822b48c6-2e86-45a3-af80-ef8cd9cf5674-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:31.155324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.155322 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/822b48c6-2e86-45a3-af80-ef8cd9cf5674-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:31.155533 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.155336 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8546\" (UniqueName: \"kubernetes.io/projected/822b48c6-2e86-45a3-af80-ef8cd9cf5674-kube-api-access-q8546\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:31.742068 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.742037 5938 generic.go:356] "Generic (PLEG): container finished" podID="822b48c6-2e86-45a3-af80-ef8cd9cf5674" containerID="3ba907c858e6fa8cc1dea62b5656d6e9c65975a4872d8797b17efff5590da984" exitCode=0 Oct 30 09:37:31.742515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.742083 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" Oct 30 09:37:31.742515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.742130 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" event={"ID":"822b48c6-2e86-45a3-af80-ef8cd9cf5674","Type":"ContainerDied","Data":"3ba907c858e6fa8cc1dea62b5656d6e9c65975a4872d8797b17efff5590da984"} Oct 30 09:37:31.742515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.742172 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" event={"ID":"822b48c6-2e86-45a3-af80-ef8cd9cf5674","Type":"ContainerDied","Data":"1fffbbca89f0dce3cbc16fe414acfb35a848fa24eb308ef4ff736e83656f1ef8"} Oct 30 09:37:31.742515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.742195 5938 scope.go:122] "RemoveContainer" containerID="3ba907c858e6fa8cc1dea62b5656d6e9c65975a4872d8797b17efff5590da984" Oct 30 09:37:31.755206 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.755182 5938 scope.go:122] "RemoveContainer" containerID="3ba907c858e6fa8cc1dea62b5656d6e9c65975a4872d8797b17efff5590da984" Oct 30 09:37:31.755521 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:37:31.755499 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba907c858e6fa8cc1dea62b5656d6e9c65975a4872d8797b17efff5590da984\": container with ID starting with 3ba907c858e6fa8cc1dea62b5656d6e9c65975a4872d8797b17efff5590da984 not found: ID does not exist" containerID="3ba907c858e6fa8cc1dea62b5656d6e9c65975a4872d8797b17efff5590da984" Oct 30 09:37:31.755561 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.755532 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba907c858e6fa8cc1dea62b5656d6e9c65975a4872d8797b17efff5590da984"} err="failed to get container status \"3ba907c858e6fa8cc1dea62b5656d6e9c65975a4872d8797b17efff5590da984\": rpc error: code = NotFound desc = could not find container \"3ba907c858e6fa8cc1dea62b5656d6e9c65975a4872d8797b17efff5590da984\": container with ID starting with 3ba907c858e6fa8cc1dea62b5656d6e9c65975a4872d8797b17efff5590da984 not found: ID does not exist" Oct 30 09:37:31.802203 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.802129 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns"] Oct 30 09:37:31.816296 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.816256 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns"] Oct 30 09:37:31.870792 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:31.870748 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-7c88ae/auth-test-model-7c88ae-kserve-8898c9765-pxjns" podUID="822b48c6-2e86-45a3-af80-ef8cd9cf5674" containerName="main" probeResult="failure" output="Get \"https://10.134.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:37:32.259301 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:32.259271 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="822b48c6-2e86-45a3-af80-ef8cd9cf5674" path="/var/lib/kubelet/pods/822b48c6-2e86-45a3-af80-ef8cd9cf5674/volumes" Oct 30 09:37:33.739454 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.739419 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d"] Oct 30 09:37:33.740108 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.740092 5938 cpu_manager.go:414] "containerMap: removing container" podUID="822b48c6-2e86-45a3-af80-ef8cd9cf5674" containerName="main" Oct 30 09:37:33.740163 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.740110 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822b48c6-2e86-45a3-af80-ef8cd9cf5674" containerName="main" Oct 30 09:37:33.740163 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.740147 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a61b1d50-c810-4bcb-a406-f884a8f758f9" containerName="revoke-keys" Oct 30 09:37:33.740163 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.740152 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a61b1d50-c810-4bcb-a406-f884a8f758f9" containerName="revoke-keys" Oct 30 09:37:33.740163 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.740162 5938 cpu_manager.go:414] "containerMap: removing container" podUID="957b9998-b762-4673-bf29-1d5b8d7ae904" containerName="main" Oct 30 09:37:33.740291 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.740167 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957b9998-b762-4673-bf29-1d5b8d7ae904" containerName="main" Oct 30 09:37:33.740291 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.740264 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="822b48c6-2e86-45a3-af80-ef8cd9cf5674" containerName="main" Oct 30 09:37:33.740291 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.740278 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a61b1d50-c810-4bcb-a406-f884a8f758f9" containerName="revoke-keys" Oct 30 09:37:33.740291 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.740284 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="957b9998-b762-4673-bf29-1d5b8d7ae904" containerName="main" Oct 30 09:37:33.745774 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.745754 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" Oct 30 09:37:33.763738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.763703 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d"] Oct 30 09:37:33.774266 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.774229 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ae795b\"" Oct 30 09:37:33.774266 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.774235 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ae795b-dockercfg-2ccnp\"" Oct 30 09:37:33.860179 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.860125 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4"] Oct 30 09:37:33.893526 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.893490 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrwh8\" (UniqueName: \"kubernetes.io/projected/03aa9f18-97ff-48ed-9aa0-fcea685be2b2-kube-api-access-rrwh8\") pod \"payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d\" (UID: \"03aa9f18-97ff-48ed-9aa0-fcea685be2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" Oct 30 09:37:33.893939 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.893918 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03aa9f18-97ff-48ed-9aa0-fcea685be2b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d\" (UID: \"03aa9f18-97ff-48ed-9aa0-fcea685be2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" Oct 30 09:37:33.921269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.921238 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4"] Oct 30 09:37:33.921930 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.921852 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" Oct 30 09:37:33.995563 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.995484 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrwh8\" (UniqueName: \"kubernetes.io/projected/03aa9f18-97ff-48ed-9aa0-fcea685be2b2-kube-api-access-rrwh8\") pod \"payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d\" (UID: \"03aa9f18-97ff-48ed-9aa0-fcea685be2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" Oct 30 09:37:33.995791 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.995635 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03aa9f18-97ff-48ed-9aa0-fcea685be2b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d\" (UID: \"03aa9f18-97ff-48ed-9aa0-fcea685be2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" Oct 30 09:37:33.996375 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:33.996348 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03aa9f18-97ff-48ed-9aa0-fcea685be2b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d\" (UID: \"03aa9f18-97ff-48ed-9aa0-fcea685be2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" Oct 30 09:37:34.016784 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.016762 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrwh8\" (UniqueName: \"kubernetes.io/projected/03aa9f18-97ff-48ed-9aa0-fcea685be2b2-kube-api-access-rrwh8\") pod \"payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d\" (UID: \"03aa9f18-97ff-48ed-9aa0-fcea685be2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" Oct 30 09:37:34.061023 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.060986 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" Oct 30 09:37:34.097836 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.097801 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfr62\" (UniqueName: \"kubernetes.io/projected/22d5adc2-3e45-4196-b59f-e2a585cb0714-kube-api-access-sfr62\") pod \"payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4\" (UID: \"22d5adc2-3e45-4196-b59f-e2a585cb0714\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" Oct 30 09:37:34.098022 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.097868 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d5adc2-3e45-4196-b59f-e2a585cb0714-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4\" (UID: \"22d5adc2-3e45-4196-b59f-e2a585cb0714\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" Oct 30 09:37:34.198936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.198904 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfr62\" (UniqueName: \"kubernetes.io/projected/22d5adc2-3e45-4196-b59f-e2a585cb0714-kube-api-access-sfr62\") pod \"payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4\" (UID: \"22d5adc2-3e45-4196-b59f-e2a585cb0714\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" Oct 30 09:37:34.199134 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.198951 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d5adc2-3e45-4196-b59f-e2a585cb0714-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4\" (UID: \"22d5adc2-3e45-4196-b59f-e2a585cb0714\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" Oct 30 09:37:34.199777 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.199751 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d5adc2-3e45-4196-b59f-e2a585cb0714-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4\" (UID: \"22d5adc2-3e45-4196-b59f-e2a585cb0714\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" Oct 30 09:37:34.219624 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.219601 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfr62\" (UniqueName: \"kubernetes.io/projected/22d5adc2-3e45-4196-b59f-e2a585cb0714-kube-api-access-sfr62\") pod \"payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4\" (UID: \"22d5adc2-3e45-4196-b59f-e2a585cb0714\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" Oct 30 09:37:34.261969 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.261909 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" Oct 30 09:37:34.372510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.371707 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d"] Oct 30 09:37:34.375754 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:37:34.375728 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03aa9f18_97ff_48ed_9aa0_fcea685be2b2.slice/crio-f0a314550d1b96699a376453a52b44ef74964370f219ac12a520a9860435d5ff WatchSource:0}: Error finding container f0a314550d1b96699a376453a52b44ef74964370f219ac12a520a9860435d5ff: Status 404 returned error can't find the container with id f0a314550d1b96699a376453a52b44ef74964370f219ac12a520a9860435d5ff Oct 30 09:37:34.566669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.566645 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4"] Oct 30 09:37:34.568264 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:37:34.568240 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d5adc2_3e45_4196_b59f_e2a585cb0714.slice/crio-3c43d00aa4fd2c003e0396e3ad08d017c0afc5298cf3157afd7f40bd2af2d644 WatchSource:0}: Error finding container 3c43d00aa4fd2c003e0396e3ad08d017c0afc5298cf3157afd7f40bd2af2d644: Status 404 returned error can't find the container with id 3c43d00aa4fd2c003e0396e3ad08d017c0afc5298cf3157afd7f40bd2af2d644 Oct 30 09:37:34.768923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.768875 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" event={"ID":"22d5adc2-3e45-4196-b59f-e2a585cb0714","Type":"ContainerStarted","Data":"fe2ccea71296a9ae269d50f4c6867fd5a42d439030131e492d0f4e4ba32bf7df"} Oct 30 09:37:34.769389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.768941 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" event={"ID":"22d5adc2-3e45-4196-b59f-e2a585cb0714","Type":"ContainerStarted","Data":"3c43d00aa4fd2c003e0396e3ad08d017c0afc5298cf3157afd7f40bd2af2d644"} Oct 30 09:37:34.769389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.768971 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" Oct 30 09:37:34.770723 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.770692 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" event={"ID":"03aa9f18-97ff-48ed-9aa0-fcea685be2b2","Type":"ContainerStarted","Data":"d0a01eded410cc016a7287c31bd2e55ddbb143b29127488411d85c2ba4145555"} Oct 30 09:37:34.770876 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.770745 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" event={"ID":"03aa9f18-97ff-48ed-9aa0-fcea685be2b2","Type":"ContainerStarted","Data":"f0a314550d1b96699a376453a52b44ef74964370f219ac12a520a9860435d5ff"} Oct 30 09:37:34.770876 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.770777 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" Oct 30 09:37:34.810456 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.810409 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" podStartSLOduration=1.8103458190000001 podStartE2EDuration="1.810345819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:37:34.809648048 +0000 UTC m=+2572.672353871" watchObservedRunningTime="2026-10-30 09:37:34.810345819 +0000 UTC m=+2572.673051641" Oct 30 09:37:34.853813 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:34.853709 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" podStartSLOduration=1.853694672 podStartE2EDuration="1.853694672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:37:34.850205515 +0000 UTC m=+2572.712911336" watchObservedRunningTime="2026-10-30 09:37:34.853694672 +0000 UTC m=+2572.716400523" Oct 30 09:37:43.348941 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:43.348910 5938 scope.go:122] "RemoveContainer" containerID="f5b0426e20b0c8162b57643237c549ab23f7fc93290901bb037ee15d6417792e" Oct 30 09:37:43.359495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:43.359469 5938 scope.go:122] "RemoveContainer" containerID="71a900597b1f0d399daf7c1768542207ec21a3d8a0dc10cce627dbfbdbe4afb4" Oct 30 09:37:43.368843 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:43.368823 5938 scope.go:122] "RemoveContainer" containerID="34b5d033f094e28c45e93f8e4be855b236bfad41baa7b390dd82692f75a755da" Oct 30 09:37:43.378022 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:43.377999 5938 scope.go:122] "RemoveContainer" containerID="c371df455f76fe5c124c875a439404860e49dd8f711db66e9dbdb7aa5b726492" Oct 30 09:37:43.387701 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:43.387681 5938 scope.go:122] "RemoveContainer" containerID="7b88773692a770eea3eb3daf0d51a931722774d79bebdbf6a814190f83680200" Oct 30 09:37:43.397835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:43.397812 5938 scope.go:122] "RemoveContainer" containerID="6853c926b9601b584eff1424dbe1997c1e471578b681399c50853e39efe22249" Oct 30 09:37:43.407971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:43.407944 5938 scope.go:122] "RemoveContainer" containerID="0d4475200c7082991b306f2be29de6d210704ac2495f1841237d8695889a0ce1" Oct 30 09:37:43.417364 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:43.417340 5938 scope.go:122] "RemoveContainer" containerID="06f7df41f0a1a397337be80f612ee5f9e97759757b8c7a19e3cc39d71f2bc5c8" Oct 30 09:37:45.782372 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:45.782337 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" Oct 30 09:37:45.783145 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:45.783099 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" Oct 30 09:37:45.809856 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:37:45.809824 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:37:49.673369 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:49.673326 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk"] Oct 30 09:37:49.680312 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:49.680285 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" Oct 30 09:37:49.696619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:49.696576 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk"] Oct 30 09:37:49.706092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:49.706050 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:37:49.706092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:49.706064 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:37:49.830340 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:49.830300 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15d9dfaf-928b-4ffe-9a11-4d859f755710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab17a479-lvzvk\" (UID: \"15d9dfaf-928b-4ffe-9a11-4d859f755710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" Oct 30 09:37:49.830526 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:49.830453 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ncjt\" (UniqueName: \"kubernetes.io/projected/15d9dfaf-928b-4ffe-9a11-4d859f755710-kube-api-access-6ncjt\") pod \"maas-api-revoke-sub-ab17a479-lvzvk\" (UID: \"15d9dfaf-928b-4ffe-9a11-4d859f755710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" Oct 30 09:37:49.830526 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:49.830517 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15d9dfaf-928b-4ffe-9a11-4d859f755710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab17a479-lvzvk\" (UID: \"15d9dfaf-928b-4ffe-9a11-4d859f755710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" Oct 30 09:37:49.932394 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:49.932305 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15d9dfaf-928b-4ffe-9a11-4d859f755710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab17a479-lvzvk\" (UID: \"15d9dfaf-928b-4ffe-9a11-4d859f755710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" Oct 30 09:37:49.932814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:49.932572 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15d9dfaf-928b-4ffe-9a11-4d859f755710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab17a479-lvzvk\" (UID: \"15d9dfaf-928b-4ffe-9a11-4d859f755710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" Oct 30 09:37:49.932814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:49.932730 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ncjt\" (UniqueName: \"kubernetes.io/projected/15d9dfaf-928b-4ffe-9a11-4d859f755710-kube-api-access-6ncjt\") pod \"maas-api-revoke-sub-ab17a479-lvzvk\" (UID: \"15d9dfaf-928b-4ffe-9a11-4d859f755710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" Oct 30 09:37:49.933322 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:49.933295 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15d9dfaf-928b-4ffe-9a11-4d859f755710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab17a479-lvzvk\" (UID: \"15d9dfaf-928b-4ffe-9a11-4d859f755710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" Oct 30 09:37:49.935601 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:49.935548 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15d9dfaf-928b-4ffe-9a11-4d859f755710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab17a479-lvzvk\" (UID: \"15d9dfaf-928b-4ffe-9a11-4d859f755710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" Oct 30 09:37:49.953856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:49.953825 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ncjt\" (UniqueName: \"kubernetes.io/projected/15d9dfaf-928b-4ffe-9a11-4d859f755710-kube-api-access-6ncjt\") pod \"maas-api-revoke-sub-ab17a479-lvzvk\" (UID: \"15d9dfaf-928b-4ffe-9a11-4d859f755710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" Oct 30 09:37:49.997092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:49.997057 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" Oct 30 09:37:50.303634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:50.303530 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk"] Oct 30 09:37:50.308402 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:37:50.308373 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d9dfaf_928b_4ffe_9a11_4d859f755710.slice/crio-5db26c5c28143db3406f742e12a2f85b95f65403fb905c02565df895854c9591 WatchSource:0}: Error finding container 5db26c5c28143db3406f742e12a2f85b95f65403fb905c02565df895854c9591: Status 404 returned error can't find the container with id 5db26c5c28143db3406f742e12a2f85b95f65403fb905c02565df895854c9591 Oct 30 09:37:50.905434 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:50.905345 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" event={"ID":"15d9dfaf-928b-4ffe-9a11-4d859f755710","Type":"ContainerStarted","Data":"48b3a696e85d462d9fac1306f49e404664873cb748e3692bbf9d51119c8cf43c"} Oct 30 09:37:50.905434 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:50.905402 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" event={"ID":"15d9dfaf-928b-4ffe-9a11-4d859f755710","Type":"ContainerStarted","Data":"5db26c5c28143db3406f742e12a2f85b95f65403fb905c02565df895854c9591"} Oct 30 09:37:50.949428 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:50.949372 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" podStartSLOduration=1.9493582379999999 podStartE2EDuration="1.949358238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:37:50.947719449 +0000 UTC m=+2588.810425275" watchObservedRunningTime="2026-10-30 09:37:50.949358238 +0000 UTC m=+2588.812064106" Oct 30 09:37:51.116083 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.116047 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4"] Oct 30 09:37:51.120963 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.120939 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4" Oct 30 09:37:51.138645 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.138616 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4"] Oct 30 09:37:51.253027 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.252993 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfb90b3a-ckhj4\" (UID: \"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4" Oct 30 09:37:51.253260 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.253233 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfb90b3a-ckhj4\" (UID: \"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4" Oct 30 09:37:51.253349 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.253290 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwsdm\" (UniqueName: \"kubernetes.io/projected/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-kube-api-access-lwsdm\") pod \"maas-api-revoke-sub-dfb90b3a-ckhj4\" (UID: \"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4" Oct 30 09:37:51.354808 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.354770 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfb90b3a-ckhj4\" (UID: \"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4" Oct 30 09:37:51.355011 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.354897 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfb90b3a-ckhj4\" (UID: \"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4" Oct 30 09:37:51.355011 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.354924 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwsdm\" (UniqueName: \"kubernetes.io/projected/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-kube-api-access-lwsdm\") pod \"maas-api-revoke-sub-dfb90b3a-ckhj4\" (UID: \"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4" Oct 30 09:37:51.355687 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.355659 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfb90b3a-ckhj4\" (UID: \"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4" Oct 30 09:37:51.357428 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.357409 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfb90b3a-ckhj4\" (UID: \"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4" Oct 30 09:37:51.376553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.376521 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwsdm\" (UniqueName: \"kubernetes.io/projected/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-kube-api-access-lwsdm\") pod \"maas-api-revoke-sub-dfb90b3a-ckhj4\" (UID: \"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4" Oct 30 09:37:51.436571 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.436542 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4" Oct 30 09:37:51.741072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.741048 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4"] Oct 30 09:37:51.742068 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:37:51.742046 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee04d9d8_c7b3_409b_bbcd_b7137d417cf2.slice/crio-a63a28f85e5152acee9917c81af0263d9effe74a0484c9ea1f34239feb4b5c28 WatchSource:0}: Error finding container a63a28f85e5152acee9917c81af0263d9effe74a0484c9ea1f34239feb4b5c28: Status 404 returned error can't find the container with id a63a28f85e5152acee9917c81af0263d9effe74a0484c9ea1f34239feb4b5c28 Oct 30 09:37:51.913831 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.913793 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4" event={"ID":"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2","Type":"ContainerStarted","Data":"a63a28f85e5152acee9917c81af0263d9effe74a0484c9ea1f34239feb4b5c28"} Oct 30 09:37:51.915658 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.915625 5938 generic.go:356] "Generic (PLEG): container finished" podID="15d9dfaf-928b-4ffe-9a11-4d859f755710" containerID="48b3a696e85d462d9fac1306f49e404664873cb748e3692bbf9d51119c8cf43c" exitCode=0 Oct 30 09:37:51.915794 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:51.915671 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" event={"ID":"15d9dfaf-928b-4ffe-9a11-4d859f755710","Type":"ContainerDied","Data":"48b3a696e85d462d9fac1306f49e404664873cb748e3692bbf9d51119c8cf43c"} Oct 30 09:37:52.924366 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:52.924328 5938 generic.go:356] "Generic (PLEG): container finished" podID="ee04d9d8-c7b3-409b-bbcd-b7137d417cf2" containerID="a5ae2fab8567cadbd7b7715b0ba5b69e2cfcb9150a46ecad5f0bc2c64b43a1bd" exitCode=0 Oct 30 09:37:52.924838 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:52.924432 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4" event={"ID":"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2","Type":"ContainerDied","Data":"a5ae2fab8567cadbd7b7715b0ba5b69e2cfcb9150a46ecad5f0bc2c64b43a1bd"} Oct 30 09:37:53.075750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:53.075725 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" Oct 30 09:37:53.178789 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:53.178758 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15d9dfaf-928b-4ffe-9a11-4d859f755710-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15d9dfaf-928b-4ffe-9a11-4d859f755710-maas-api-xks-ca\") pod \"15d9dfaf-928b-4ffe-9a11-4d859f755710\" (UID: \"15d9dfaf-928b-4ffe-9a11-4d859f755710\") " Oct 30 09:37:53.178951 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:53.178851 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d9dfaf-928b-4ffe-9a11-4d859f755710-kube-api-access-6ncjt\" (UniqueName: \"kubernetes.io/projected/15d9dfaf-928b-4ffe-9a11-4d859f755710-kube-api-access-6ncjt\") pod \"15d9dfaf-928b-4ffe-9a11-4d859f755710\" (UID: \"15d9dfaf-928b-4ffe-9a11-4d859f755710\") " Oct 30 09:37:53.179041 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:53.179023 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15d9dfaf-928b-4ffe-9a11-4d859f755710-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15d9dfaf-928b-4ffe-9a11-4d859f755710-maas-api-openshift-ca\") pod \"15d9dfaf-928b-4ffe-9a11-4d859f755710\" (UID: \"15d9dfaf-928b-4ffe-9a11-4d859f755710\") " Oct 30 09:37:53.179376 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:53.179340 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15d9dfaf-928b-4ffe-9a11-4d859f755710-maas-api-openshift-ca" pod "15d9dfaf-928b-4ffe-9a11-4d859f755710" (UID: "15d9dfaf-928b-4ffe-9a11-4d859f755710"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:37:53.179697 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:53.179614 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15d9dfaf-928b-4ffe-9a11-4d859f755710-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:53.181115 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:53.181091 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15d9dfaf-928b-4ffe-9a11-4d859f755710-maas-api-xks-ca" pod "15d9dfaf-928b-4ffe-9a11-4d859f755710" (UID: "15d9dfaf-928b-4ffe-9a11-4d859f755710"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:37:53.181258 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:53.181239 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d9dfaf-928b-4ffe-9a11-4d859f755710-kube-api-access-6ncjt" pod "15d9dfaf-928b-4ffe-9a11-4d859f755710" (UID: "15d9dfaf-928b-4ffe-9a11-4d859f755710"). InnerVolumeSpecName "kube-api-access-6ncjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:53.281477 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:53.281395 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15d9dfaf-928b-4ffe-9a11-4d859f755710-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:53.281477 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:53.281422 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ncjt\" (UniqueName: \"kubernetes.io/projected/15d9dfaf-928b-4ffe-9a11-4d859f755710-kube-api-access-6ncjt\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:53.938505 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:53.938473 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" Oct 30 09:37:53.939127 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:53.938472 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk" event={"ID":"15d9dfaf-928b-4ffe-9a11-4d859f755710","Type":"ContainerDied","Data":"5db26c5c28143db3406f742e12a2f85b95f65403fb905c02565df895854c9591"} Oct 30 09:37:53.939127 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:53.938684 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5db26c5c28143db3406f742e12a2f85b95f65403fb905c02565df895854c9591" Oct 30 09:37:54.105139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:54.105113 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4" Oct 30 09:37:54.193086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:54.193001 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-maas-api-xks-ca\") pod \"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2\" (UID: \"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2\") " Oct 30 09:37:54.193269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:54.193147 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-maas-api-openshift-ca\") pod \"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2\" (UID: \"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2\") " Oct 30 09:37:54.193269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:54.193259 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-kube-api-access-lwsdm\" (UniqueName: \"kubernetes.io/projected/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-kube-api-access-lwsdm\") pod \"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2\" (UID: \"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2\") " Oct 30 09:37:54.193493 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:54.193468 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-maas-api-openshift-ca" pod "ee04d9d8-c7b3-409b-bbcd-b7137d417cf2" (UID: "ee04d9d8-c7b3-409b-bbcd-b7137d417cf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:37:54.193869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:54.193852 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:54.195380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:54.195350 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-maas-api-xks-ca" pod "ee04d9d8-c7b3-409b-bbcd-b7137d417cf2" (UID: "ee04d9d8-c7b3-409b-bbcd-b7137d417cf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:37:54.195380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:54.195367 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-kube-api-access-lwsdm" pod "ee04d9d8-c7b3-409b-bbcd-b7137d417cf2" (UID: "ee04d9d8-c7b3-409b-bbcd-b7137d417cf2"). InnerVolumeSpecName "kube-api-access-lwsdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:54.295028 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:54.295001 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:54.295028 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:54.295027 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwsdm\" (UniqueName: \"kubernetes.io/projected/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2-kube-api-access-lwsdm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:37:54.947122 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:54.947084 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4" event={"ID":"ee04d9d8-c7b3-409b-bbcd-b7137d417cf2","Type":"ContainerDied","Data":"a63a28f85e5152acee9917c81af0263d9effe74a0484c9ea1f34239feb4b5c28"} Oct 30 09:37:54.947482 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:54.947134 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4" Oct 30 09:37:54.947482 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:37:54.947139 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a63a28f85e5152acee9917c81af0263d9effe74a0484c9ea1f34239feb4b5c28" Oct 30 09:38:00.005323 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.005288 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l"] Oct 30 09:38:00.005923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.005909 5938 cpu_manager.go:414] "containerMap: removing container" podUID="15d9dfaf-928b-4ffe-9a11-4d859f755710" containerName="revoke-keys" Oct 30 09:38:00.005972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.005924 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15d9dfaf-928b-4ffe-9a11-4d859f755710" containerName="revoke-keys" Oct 30 09:38:00.005972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.005955 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ee04d9d8-c7b3-409b-bbcd-b7137d417cf2" containerName="revoke-keys" Oct 30 09:38:00.005972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.005961 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee04d9d8-c7b3-409b-bbcd-b7137d417cf2" containerName="revoke-keys" Oct 30 09:38:00.006114 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.006056 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee04d9d8-c7b3-409b-bbcd-b7137d417cf2" containerName="revoke-keys" Oct 30 09:38:00.006114 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.006072 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="15d9dfaf-928b-4ffe-9a11-4d859f755710" containerName="revoke-keys" Oct 30 09:38:00.010003 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.009987 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" Oct 30 09:38:00.027850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.027821 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l"] Oct 30 09:38:00.052007 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.049974 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:38:00.052007 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.050127 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:38:00.091239 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.091143 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj5cv\" (UniqueName: \"kubernetes.io/projected/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-kube-api-access-gj5cv\") pod \"maas-api-revoke-sub-882b02de-zld9l\" (UID: \"6cb11947-f2ef-4a23-880e-17f2cccf7c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" Oct 30 09:38:00.091429 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.091286 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-882b02de-zld9l\" (UID: \"6cb11947-f2ef-4a23-880e-17f2cccf7c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" Oct 30 09:38:00.091429 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.091329 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-882b02de-zld9l\" (UID: \"6cb11947-f2ef-4a23-880e-17f2cccf7c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" Oct 30 09:38:00.193173 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.193141 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj5cv\" (UniqueName: \"kubernetes.io/projected/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-kube-api-access-gj5cv\") pod \"maas-api-revoke-sub-882b02de-zld9l\" (UID: \"6cb11947-f2ef-4a23-880e-17f2cccf7c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" Oct 30 09:38:00.193345 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.193250 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-882b02de-zld9l\" (UID: \"6cb11947-f2ef-4a23-880e-17f2cccf7c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" Oct 30 09:38:00.193345 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.193284 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-882b02de-zld9l\" (UID: \"6cb11947-f2ef-4a23-880e-17f2cccf7c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" Oct 30 09:38:00.193974 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.193949 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-882b02de-zld9l\" (UID: \"6cb11947-f2ef-4a23-880e-17f2cccf7c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" Oct 30 09:38:00.195831 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.195806 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-882b02de-zld9l\" (UID: \"6cb11947-f2ef-4a23-880e-17f2cccf7c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" Oct 30 09:38:00.215920 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.215890 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj5cv\" (UniqueName: \"kubernetes.io/projected/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-kube-api-access-gj5cv\") pod \"maas-api-revoke-sub-882b02de-zld9l\" (UID: \"6cb11947-f2ef-4a23-880e-17f2cccf7c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" Oct 30 09:38:00.325050 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.324970 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" Oct 30 09:38:00.627008 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.626982 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l"] Oct 30 09:38:00.628145 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:38:00.628119 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb11947_f2ef_4a23_880e_17f2cccf7c0e.slice/crio-ed42c2aeba74a6c4af4f1efc4699e4bf2dbac7ddba23b7306b64b3317a79a486 WatchSource:0}: Error finding container ed42c2aeba74a6c4af4f1efc4699e4bf2dbac7ddba23b7306b64b3317a79a486: Status 404 returned error can't find the container with id ed42c2aeba74a6c4af4f1efc4699e4bf2dbac7ddba23b7306b64b3317a79a486 Oct 30 09:38:00.991877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.991844 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" event={"ID":"6cb11947-f2ef-4a23-880e-17f2cccf7c0e","Type":"ContainerStarted","Data":"3f4506ec3143315f0165dc821e21ee44f6490cc60e6fe9c9bd0d8b91d2425361"} Oct 30 09:38:00.992031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:00.991891 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" event={"ID":"6cb11947-f2ef-4a23-880e-17f2cccf7c0e","Type":"ContainerStarted","Data":"ed42c2aeba74a6c4af4f1efc4699e4bf2dbac7ddba23b7306b64b3317a79a486"} Oct 30 09:38:01.034930 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:01.034881 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" podStartSLOduration=2.034865009 podStartE2EDuration="2.034865009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:01.031567634 +0000 UTC m=+2598.894273458" watchObservedRunningTime="2026-10-30 09:38:01.034865009 +0000 UTC m=+2598.897570859" Oct 30 09:38:01.456794 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:01.456728 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852"] Oct 30 09:38:01.463527 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:01.463502 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" Oct 30 09:38:01.476983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:01.476958 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852"] Oct 30 09:38:01.513500 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:01.513471 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7kdv\" (UniqueName: \"kubernetes.io/projected/145760b0-8e56-4019-ad04-f0aedf4265be-kube-api-access-l7kdv\") pod \"maas-api-revoke-sub-a7037b5f-57852\" (UID: \"145760b0-8e56-4019-ad04-f0aedf4265be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" Oct 30 09:38:01.513742 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:01.513720 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145760b0-8e56-4019-ad04-f0aedf4265be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7037b5f-57852\" (UID: \"145760b0-8e56-4019-ad04-f0aedf4265be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" Oct 30 09:38:01.513793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:01.513764 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145760b0-8e56-4019-ad04-f0aedf4265be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7037b5f-57852\" (UID: \"145760b0-8e56-4019-ad04-f0aedf4265be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" Oct 30 09:38:01.615399 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:01.615372 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7kdv\" (UniqueName: \"kubernetes.io/projected/145760b0-8e56-4019-ad04-f0aedf4265be-kube-api-access-l7kdv\") pod \"maas-api-revoke-sub-a7037b5f-57852\" (UID: \"145760b0-8e56-4019-ad04-f0aedf4265be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" Oct 30 09:38:01.615542 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:01.615479 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145760b0-8e56-4019-ad04-f0aedf4265be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7037b5f-57852\" (UID: \"145760b0-8e56-4019-ad04-f0aedf4265be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" Oct 30 09:38:01.615542 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:01.615500 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145760b0-8e56-4019-ad04-f0aedf4265be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7037b5f-57852\" (UID: \"145760b0-8e56-4019-ad04-f0aedf4265be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" Oct 30 09:38:01.616128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:01.616105 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145760b0-8e56-4019-ad04-f0aedf4265be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7037b5f-57852\" (UID: \"145760b0-8e56-4019-ad04-f0aedf4265be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" Oct 30 09:38:01.617962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:01.617941 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145760b0-8e56-4019-ad04-f0aedf4265be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7037b5f-57852\" (UID: \"145760b0-8e56-4019-ad04-f0aedf4265be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" Oct 30 09:38:01.636280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:01.636255 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7kdv\" (UniqueName: \"kubernetes.io/projected/145760b0-8e56-4019-ad04-f0aedf4265be-kube-api-access-l7kdv\") pod \"maas-api-revoke-sub-a7037b5f-57852\" (UID: \"145760b0-8e56-4019-ad04-f0aedf4265be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" Oct 30 09:38:01.779760 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:01.779670 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" Oct 30 09:38:02.001021 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:02.000986 5938 generic.go:356] "Generic (PLEG): container finished" podID="6cb11947-f2ef-4a23-880e-17f2cccf7c0e" containerID="3f4506ec3143315f0165dc821e21ee44f6490cc60e6fe9c9bd0d8b91d2425361" exitCode=0 Oct 30 09:38:02.001189 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:02.001072 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" event={"ID":"6cb11947-f2ef-4a23-880e-17f2cccf7c0e","Type":"ContainerDied","Data":"3f4506ec3143315f0165dc821e21ee44f6490cc60e6fe9c9bd0d8b91d2425361"} Oct 30 09:38:02.084045 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:02.084018 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852"] Oct 30 09:38:02.084903 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:38:02.084875 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145760b0_8e56_4019_ad04_f0aedf4265be.slice/crio-52068eafdf8b6849fdec8cd9d0a459ea5958dbf0cf42b4aedfcbe0e1aeec0c31 WatchSource:0}: Error finding container 52068eafdf8b6849fdec8cd9d0a459ea5958dbf0cf42b4aedfcbe0e1aeec0c31: Status 404 returned error can't find the container with id 52068eafdf8b6849fdec8cd9d0a459ea5958dbf0cf42b4aedfcbe0e1aeec0c31 Oct 30 09:38:03.009955 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:03.009922 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" event={"ID":"145760b0-8e56-4019-ad04-f0aedf4265be","Type":"ContainerStarted","Data":"8699d1254cde90b303c134d5e86dd61783a6442912fdfb49475641bbfd26febf"} Oct 30 09:38:03.010162 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:03.009977 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" event={"ID":"145760b0-8e56-4019-ad04-f0aedf4265be","Type":"ContainerStarted","Data":"52068eafdf8b6849fdec8cd9d0a459ea5958dbf0cf42b4aedfcbe0e1aeec0c31"} Oct 30 09:38:03.053299 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:03.053248 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" podStartSLOduration=2.053235534 podStartE2EDuration="2.053235534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:03.051008669 +0000 UTC m=+2600.913714494" watchObservedRunningTime="2026-10-30 09:38:03.053235534 +0000 UTC m=+2600.915941354" Oct 30 09:38:03.181903 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:03.181877 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" Oct 30 09:38:03.240872 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:03.240841 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-kube-api-access-gj5cv\" (UniqueName: \"kubernetes.io/projected/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-kube-api-access-gj5cv\") pod \"6cb11947-f2ef-4a23-880e-17f2cccf7c0e\" (UID: \"6cb11947-f2ef-4a23-880e-17f2cccf7c0e\") " Oct 30 09:38:03.241069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:03.240920 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-maas-api-xks-ca\") pod \"6cb11947-f2ef-4a23-880e-17f2cccf7c0e\" (UID: \"6cb11947-f2ef-4a23-880e-17f2cccf7c0e\") " Oct 30 09:38:03.241069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:03.240991 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-maas-api-openshift-ca\") pod \"6cb11947-f2ef-4a23-880e-17f2cccf7c0e\" (UID: \"6cb11947-f2ef-4a23-880e-17f2cccf7c0e\") " Oct 30 09:38:03.241398 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:03.241367 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-maas-api-openshift-ca" pod "6cb11947-f2ef-4a23-880e-17f2cccf7c0e" (UID: "6cb11947-f2ef-4a23-880e-17f2cccf7c0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:38:03.241697 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:03.241671 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:03.244008 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:03.243972 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-kube-api-access-gj5cv" pod "6cb11947-f2ef-4a23-880e-17f2cccf7c0e" (UID: "6cb11947-f2ef-4a23-880e-17f2cccf7c0e"). InnerVolumeSpecName "kube-api-access-gj5cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:03.244389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:03.244366 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-maas-api-xks-ca" pod "6cb11947-f2ef-4a23-880e-17f2cccf7c0e" (UID: "6cb11947-f2ef-4a23-880e-17f2cccf7c0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:38:03.342819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:03.342742 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj5cv\" (UniqueName: \"kubernetes.io/projected/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-kube-api-access-gj5cv\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:03.342819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:03.342767 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb11947-f2ef-4a23-880e-17f2cccf7c0e-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:04.023628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:04.020603 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" Oct 30 09:38:04.023628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:04.020635 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l" event={"ID":"6cb11947-f2ef-4a23-880e-17f2cccf7c0e","Type":"ContainerDied","Data":"ed42c2aeba74a6c4af4f1efc4699e4bf2dbac7ddba23b7306b64b3317a79a486"} Oct 30 09:38:04.023628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:04.020697 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed42c2aeba74a6c4af4f1efc4699e4bf2dbac7ddba23b7306b64b3317a79a486" Oct 30 09:38:04.023628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:04.023500 5938 generic.go:356] "Generic (PLEG): container finished" podID="145760b0-8e56-4019-ad04-f0aedf4265be" containerID="8699d1254cde90b303c134d5e86dd61783a6442912fdfb49475641bbfd26febf" exitCode=0 Oct 30 09:38:04.023628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:04.023550 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" event={"ID":"145760b0-8e56-4019-ad04-f0aedf4265be","Type":"ContainerDied","Data":"8699d1254cde90b303c134d5e86dd61783a6442912fdfb49475641bbfd26febf"} Oct 30 09:38:05.196414 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:05.196391 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" Oct 30 09:38:05.277547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:05.277515 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145760b0-8e56-4019-ad04-f0aedf4265be-kube-api-access-l7kdv\" (UniqueName: \"kubernetes.io/projected/145760b0-8e56-4019-ad04-f0aedf4265be-kube-api-access-l7kdv\") pod \"145760b0-8e56-4019-ad04-f0aedf4265be\" (UID: \"145760b0-8e56-4019-ad04-f0aedf4265be\") " Oct 30 09:38:05.277753 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:05.277649 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/145760b0-8e56-4019-ad04-f0aedf4265be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145760b0-8e56-4019-ad04-f0aedf4265be-maas-api-openshift-ca\") pod \"145760b0-8e56-4019-ad04-f0aedf4265be\" (UID: \"145760b0-8e56-4019-ad04-f0aedf4265be\") " Oct 30 09:38:05.277753 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:05.277706 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/145760b0-8e56-4019-ad04-f0aedf4265be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145760b0-8e56-4019-ad04-f0aedf4265be-maas-api-xks-ca\") pod \"145760b0-8e56-4019-ad04-f0aedf4265be\" (UID: \"145760b0-8e56-4019-ad04-f0aedf4265be\") " Oct 30 09:38:05.278111 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:05.278077 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145760b0-8e56-4019-ad04-f0aedf4265be-maas-api-openshift-ca" pod "145760b0-8e56-4019-ad04-f0aedf4265be" (UID: "145760b0-8e56-4019-ad04-f0aedf4265be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:38:05.278515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:05.278492 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145760b0-8e56-4019-ad04-f0aedf4265be-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:05.280017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:05.279992 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145760b0-8e56-4019-ad04-f0aedf4265be-maas-api-xks-ca" pod "145760b0-8e56-4019-ad04-f0aedf4265be" (UID: "145760b0-8e56-4019-ad04-f0aedf4265be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:38:05.280118 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:05.280031 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145760b0-8e56-4019-ad04-f0aedf4265be-kube-api-access-l7kdv" pod "145760b0-8e56-4019-ad04-f0aedf4265be" (UID: "145760b0-8e56-4019-ad04-f0aedf4265be"). InnerVolumeSpecName "kube-api-access-l7kdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:05.380685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:05.380643 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7kdv\" (UniqueName: \"kubernetes.io/projected/145760b0-8e56-4019-ad04-f0aedf4265be-kube-api-access-l7kdv\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:05.380685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:05.380668 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145760b0-8e56-4019-ad04-f0aedf4265be-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:06.042574 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:06.042542 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" Oct 30 09:38:06.042574 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:06.042554 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852" event={"ID":"145760b0-8e56-4019-ad04-f0aedf4265be","Type":"ContainerDied","Data":"52068eafdf8b6849fdec8cd9d0a459ea5958dbf0cf42b4aedfcbe0e1aeec0c31"} Oct 30 09:38:06.042826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:06.042650 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52068eafdf8b6849fdec8cd9d0a459ea5958dbf0cf42b4aedfcbe0e1aeec0c31" Oct 30 09:38:12.208131 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:12.208095 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr"] Oct 30 09:38:14.319379 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:14.319313 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8"] Oct 30 09:38:15.809856 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:38:15.809820 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:38:22.734145 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:22.734060 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn"] Oct 30 09:38:22.734571 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:22.734263 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" podUID="b4b3c5d0-f0a6-4077-b212-724b6dbe378a" containerName="istio-proxy" containerID="cri-o://b68000275cda1ad5b64c3c52f663266e22cd1527368e29f1f1c9c141d68dae17" gracePeriod=30 Oct 30 09:38:24.833617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:24.775518 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw"] Oct 30 09:38:27.245289 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.245250 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" podUID="64267c50-7a21-4c55-aa19-f0899841dad6" containerName="main" containerID="cri-o://c1b1bef1de7824e938d771ae100f479e7e84c361a3c8160416f3c6850d18eaaa" gracePeriod=45 Oct 30 09:38:27.531519 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.531487 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:38:27.588881 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.588849 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64267c50-7a21-4c55-aa19-f0899841dad6-tls-certs\" (UniqueName: \"kubernetes.io/secret/64267c50-7a21-4c55-aa19-f0899841dad6-tls-certs\") pod \"64267c50-7a21-4c55-aa19-f0899841dad6\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " Oct 30 09:38:27.589096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.588974 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64267c50-7a21-4c55-aa19-f0899841dad6-kube-api-access-zzxzz\" (UniqueName: \"kubernetes.io/projected/64267c50-7a21-4c55-aa19-f0899841dad6-kube-api-access-zzxzz\") pod \"64267c50-7a21-4c55-aa19-f0899841dad6\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " Oct 30 09:38:27.589096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.589049 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-home\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-home\") pod \"64267c50-7a21-4c55-aa19-f0899841dad6\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " Oct 30 09:38:27.589234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.589130 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-model-cache\") pod \"64267c50-7a21-4c55-aa19-f0899841dad6\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " Oct 30 09:38:27.589234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.589175 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-tmp-dir\") pod \"64267c50-7a21-4c55-aa19-f0899841dad6\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " Oct 30 09:38:27.589234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.589211 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-dshm\") pod \"64267c50-7a21-4c55-aa19-f0899841dad6\" (UID: \"64267c50-7a21-4c55-aa19-f0899841dad6\") " Oct 30 09:38:27.589627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.589601 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-model-cache" pod "64267c50-7a21-4c55-aa19-f0899841dad6" (UID: "64267c50-7a21-4c55-aa19-f0899841dad6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:27.589741 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.589638 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-tmp-dir" pod "64267c50-7a21-4c55-aa19-f0899841dad6" (UID: "64267c50-7a21-4c55-aa19-f0899841dad6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:27.589910 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.589870 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-home" pod "64267c50-7a21-4c55-aa19-f0899841dad6" (UID: "64267c50-7a21-4c55-aa19-f0899841dad6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:27.590510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.590423 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:27.590510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.590453 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:27.590510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.590468 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:27.593175 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.593145 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-dshm" pod "64267c50-7a21-4c55-aa19-f0899841dad6" (UID: "64267c50-7a21-4c55-aa19-f0899841dad6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:27.593278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.593174 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64267c50-7a21-4c55-aa19-f0899841dad6-kube-api-access-zzxzz" pod "64267c50-7a21-4c55-aa19-f0899841dad6" (UID: "64267c50-7a21-4c55-aa19-f0899841dad6"). InnerVolumeSpecName "kube-api-access-zzxzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:27.593632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.593566 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64267c50-7a21-4c55-aa19-f0899841dad6-tls-certs" pod "64267c50-7a21-4c55-aa19-f0899841dad6" (UID: "64267c50-7a21-4c55-aa19-f0899841dad6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:38:27.692077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.692042 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzxzz\" (UniqueName: \"kubernetes.io/projected/64267c50-7a21-4c55-aa19-f0899841dad6-kube-api-access-zzxzz\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:27.692077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.692071 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64267c50-7a21-4c55-aa19-f0899841dad6-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:27.692077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.692081 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64267c50-7a21-4c55-aa19-f0899841dad6-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:27.996741 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:27.996714 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:38:28.097979 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.097890 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-podinfo\") pod \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " Oct 30 09:38:28.097979 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.097934 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-envoy\") pod \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " Oct 30 09:38:28.098230 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.097998 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-data\") pod \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " Oct 30 09:38:28.098230 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.098089 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istiod-ca-cert\") pod \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " Oct 30 09:38:28.098230 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.098219 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-credential-socket\") pod \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " Oct 30 09:38:28.098407 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.098272 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-kube-api-access-n94b9\" (UniqueName: \"kubernetes.io/projected/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-kube-api-access-n94b9\") pod \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " Oct 30 09:38:28.098407 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.098358 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-token\" (UniqueName: \"kubernetes.io/projected/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-token\") pod \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " Oct 30 09:38:28.098510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.098412 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-workload-certs\") pod \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " Oct 30 09:38:28.098510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.098471 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-workload-socket\") pod \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\" (UID: \"b4b3c5d0-f0a6-4077-b212-724b6dbe378a\") " Oct 30 09:38:28.098630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.098543 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-data" pod "b4b3c5d0-f0a6-4077-b212-724b6dbe378a" (UID: "b4b3c5d0-f0a6-4077-b212-724b6dbe378a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:28.098962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.098935 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istiod-ca-cert" pod "b4b3c5d0-f0a6-4077-b212-724b6dbe378a" (UID: "b4b3c5d0-f0a6-4077-b212-724b6dbe378a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:38:28.099117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.099098 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-workload-socket" pod "b4b3c5d0-f0a6-4077-b212-724b6dbe378a" (UID: "b4b3c5d0-f0a6-4077-b212-724b6dbe378a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:28.099324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.099295 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:28.099457 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.099324 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:28.099457 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.099353 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:28.099908 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.099575 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-credential-socket" pod "b4b3c5d0-f0a6-4077-b212-724b6dbe378a" (UID: "b4b3c5d0-f0a6-4077-b212-724b6dbe378a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:28.099908 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.099875 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-workload-certs" pod "b4b3c5d0-f0a6-4077-b212-724b6dbe378a" (UID: "b4b3c5d0-f0a6-4077-b212-724b6dbe378a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:28.101318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.101296 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-token" pod "b4b3c5d0-f0a6-4077-b212-724b6dbe378a" (UID: "b4b3c5d0-f0a6-4077-b212-724b6dbe378a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:28.101982 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.101959 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-envoy" pod "b4b3c5d0-f0a6-4077-b212-724b6dbe378a" (UID: "b4b3c5d0-f0a6-4077-b212-724b6dbe378a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:28.102850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.102830 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-podinfo" pod "b4b3c5d0-f0a6-4077-b212-724b6dbe378a" (UID: "b4b3c5d0-f0a6-4077-b212-724b6dbe378a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:38:28.103556 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.103531 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-kube-api-access-n94b9" pod "b4b3c5d0-f0a6-4077-b212-724b6dbe378a" (UID: "b4b3c5d0-f0a6-4077-b212-724b6dbe378a"). InnerVolumeSpecName "kube-api-access-n94b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:28.200724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.200697 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:28.200724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.200721 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:28.200724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.200731 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:28.200973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.200742 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:28.200973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.200751 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:28.200973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.200759 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n94b9\" (UniqueName: \"kubernetes.io/projected/b4b3c5d0-f0a6-4077-b212-724b6dbe378a-kube-api-access-n94b9\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:28.221551 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.221516 5938 generic.go:356] "Generic (PLEG): container finished" podID="64267c50-7a21-4c55-aa19-f0899841dad6" containerID="c1b1bef1de7824e938d771ae100f479e7e84c361a3c8160416f3c6850d18eaaa" exitCode=0 Oct 30 09:38:28.221736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.221573 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" Oct 30 09:38:28.221736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.221612 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" event={"ID":"64267c50-7a21-4c55-aa19-f0899841dad6","Type":"ContainerDied","Data":"c1b1bef1de7824e938d771ae100f479e7e84c361a3c8160416f3c6850d18eaaa"} Oct 30 09:38:28.221736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.221657 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr" event={"ID":"64267c50-7a21-4c55-aa19-f0899841dad6","Type":"ContainerDied","Data":"fd8129d6960634fadcf548b675e3526c6c7bd8b95f1483ff5ded9c2e5de71422"} Oct 30 09:38:28.221736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.221679 5938 scope.go:122] "RemoveContainer" containerID="c1b1bef1de7824e938d771ae100f479e7e84c361a3c8160416f3c6850d18eaaa" Oct 30 09:38:28.223332 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.223303 5938 generic.go:356] "Generic (PLEG): container finished" podID="b4b3c5d0-f0a6-4077-b212-724b6dbe378a" containerID="b68000275cda1ad5b64c3c52f663266e22cd1527368e29f1f1c9c141d68dae17" exitCode=0 Oct 30 09:38:28.223451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.223348 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" event={"ID":"b4b3c5d0-f0a6-4077-b212-724b6dbe378a","Type":"ContainerDied","Data":"b68000275cda1ad5b64c3c52f663266e22cd1527368e29f1f1c9c141d68dae17"} Oct 30 09:38:28.223451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.223432 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" event={"ID":"b4b3c5d0-f0a6-4077-b212-724b6dbe378a","Type":"ContainerDied","Data":"09ec4d634d09f21f02b8dcfc6db94b439d4d24d1f69943bb2a2795881a2ecda6"} Oct 30 09:38:28.223544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.223353 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn" Oct 30 09:38:28.233903 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.233884 5938 scope.go:122] "RemoveContainer" containerID="c1b1bef1de7824e938d771ae100f479e7e84c361a3c8160416f3c6850d18eaaa" Oct 30 09:38:28.234308 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:38:28.234277 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b1bef1de7824e938d771ae100f479e7e84c361a3c8160416f3c6850d18eaaa\": container with ID starting with c1b1bef1de7824e938d771ae100f479e7e84c361a3c8160416f3c6850d18eaaa not found: ID does not exist" containerID="c1b1bef1de7824e938d771ae100f479e7e84c361a3c8160416f3c6850d18eaaa" Oct 30 09:38:28.234410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.234320 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b1bef1de7824e938d771ae100f479e7e84c361a3c8160416f3c6850d18eaaa"} err="failed to get container status \"c1b1bef1de7824e938d771ae100f479e7e84c361a3c8160416f3c6850d18eaaa\": rpc error: code = NotFound desc = could not find container \"c1b1bef1de7824e938d771ae100f479e7e84c361a3c8160416f3c6850d18eaaa\": container with ID starting with c1b1bef1de7824e938d771ae100f479e7e84c361a3c8160416f3c6850d18eaaa not found: ID does not exist" Oct 30 09:38:28.234410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.234356 5938 scope.go:122] "RemoveContainer" containerID="b68000275cda1ad5b64c3c52f663266e22cd1527368e29f1f1c9c141d68dae17" Oct 30 09:38:28.246124 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.245919 5938 scope.go:122] "RemoveContainer" containerID="b68000275cda1ad5b64c3c52f663266e22cd1527368e29f1f1c9c141d68dae17" Oct 30 09:38:28.246471 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:38:28.246443 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68000275cda1ad5b64c3c52f663266e22cd1527368e29f1f1c9c141d68dae17\": container with ID starting with b68000275cda1ad5b64c3c52f663266e22cd1527368e29f1f1c9c141d68dae17 not found: ID does not exist" containerID="b68000275cda1ad5b64c3c52f663266e22cd1527368e29f1f1c9c141d68dae17" Oct 30 09:38:28.246614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.246481 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68000275cda1ad5b64c3c52f663266e22cd1527368e29f1f1c9c141d68dae17"} err="failed to get container status \"b68000275cda1ad5b64c3c52f663266e22cd1527368e29f1f1c9c141d68dae17\": rpc error: code = NotFound desc = could not find container \"b68000275cda1ad5b64c3c52f663266e22cd1527368e29f1f1c9c141d68dae17\": container with ID starting with b68000275cda1ad5b64c3c52f663266e22cd1527368e29f1f1c9c141d68dae17 not found: ID does not exist" Oct 30 09:38:28.277241 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.277210 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr"] Oct 30 09:38:28.280233 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.280213 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fc7ee7/auth-test-model-fc7ee7-kserve-5d7cb846b9-c4rsr"] Oct 30 09:38:28.315545 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.315513 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn"] Oct 30 09:38:28.318608 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:28.318568 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gs7bn"] Oct 30 09:38:29.345685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.345647 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" podUID="6296acea-d4ae-4735-b8ab-22f8ed8d2bd8" containerName="main" containerID="cri-o://4b872b584359fc8741c0c85c4fe173d08d74e99f298c8718bcf0c6caab53434e" gracePeriod=45 Oct 30 09:38:29.727392 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.727362 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:38:29.844867 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.844824 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-tls-certs\" (UniqueName: \"kubernetes.io/secret/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-tls-certs\") pod \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " Oct 30 09:38:29.845193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.845144 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-dshm\") pod \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " Oct 30 09:38:29.845388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.845366 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-tmp-dir\") pod \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " Oct 30 09:38:29.846497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.845868 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-kube-api-access-w7zzl\" (UniqueName: \"kubernetes.io/projected/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-kube-api-access-w7zzl\") pod \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " Oct 30 09:38:29.846497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.845918 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-model-cache\") pod \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " Oct 30 09:38:29.846497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.846108 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-home\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-home\") pod \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\" (UID: \"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8\") " Oct 30 09:38:29.850610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.847175 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-tmp-dir" pod "6296acea-d4ae-4735-b8ab-22f8ed8d2bd8" (UID: "6296acea-d4ae-4735-b8ab-22f8ed8d2bd8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:29.850610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.847711 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-model-cache" pod "6296acea-d4ae-4735-b8ab-22f8ed8d2bd8" (UID: "6296acea-d4ae-4735-b8ab-22f8ed8d2bd8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:29.850610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.848011 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-home" pod "6296acea-d4ae-4735-b8ab-22f8ed8d2bd8" (UID: "6296acea-d4ae-4735-b8ab-22f8ed8d2bd8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:29.853575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.851708 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-dshm" pod "6296acea-d4ae-4735-b8ab-22f8ed8d2bd8" (UID: "6296acea-d4ae-4735-b8ab-22f8ed8d2bd8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:29.855887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.855113 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-kube-api-access-w7zzl" pod "6296acea-d4ae-4735-b8ab-22f8ed8d2bd8" (UID: "6296acea-d4ae-4735-b8ab-22f8ed8d2bd8"). InnerVolumeSpecName "kube-api-access-w7zzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:29.855887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.855643 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:29.855887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.855666 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:29.855887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.855703 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:29.855887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.855723 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7zzl\" (UniqueName: \"kubernetes.io/projected/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-kube-api-access-w7zzl\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:29.855887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.855746 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:29.861522 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.861455 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-tls-certs" pod "6296acea-d4ae-4735-b8ab-22f8ed8d2bd8" (UID: "6296acea-d4ae-4735-b8ab-22f8ed8d2bd8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:38:29.957081 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:29.957052 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:30.244840 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:30.244800 5938 generic.go:356] "Generic (PLEG): container finished" podID="6296acea-d4ae-4735-b8ab-22f8ed8d2bd8" containerID="4b872b584359fc8741c0c85c4fe173d08d74e99f298c8718bcf0c6caab53434e" exitCode=0 Oct 30 09:38:30.245076 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:30.244874 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" Oct 30 09:38:30.245076 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:30.244887 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" event={"ID":"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8","Type":"ContainerDied","Data":"4b872b584359fc8741c0c85c4fe173d08d74e99f298c8718bcf0c6caab53434e"} Oct 30 09:38:30.245076 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:30.244934 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8" event={"ID":"6296acea-d4ae-4735-b8ab-22f8ed8d2bd8","Type":"ContainerDied","Data":"4f9046906d666bed4677c35f5bc0ad3bddfa01cfd02c0350bd49ef56af378890"} Oct 30 09:38:30.245076 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:30.244953 5938 scope.go:122] "RemoveContainer" containerID="4b872b584359fc8741c0c85c4fe173d08d74e99f298c8718bcf0c6caab53434e" Oct 30 09:38:30.262811 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:30.262786 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64267c50-7a21-4c55-aa19-f0899841dad6" path="/var/lib/kubelet/pods/64267c50-7a21-4c55-aa19-f0899841dad6/volumes" Oct 30 09:38:30.263259 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:30.263242 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4b3c5d0-f0a6-4077-b212-724b6dbe378a" path="/var/lib/kubelet/pods/b4b3c5d0-f0a6-4077-b212-724b6dbe378a/volumes" Oct 30 09:38:30.264506 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:30.264459 5938 scope.go:122] "RemoveContainer" containerID="4b872b584359fc8741c0c85c4fe173d08d74e99f298c8718bcf0c6caab53434e" Oct 30 09:38:30.264820 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:38:30.264795 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b872b584359fc8741c0c85c4fe173d08d74e99f298c8718bcf0c6caab53434e\": container with ID starting with 4b872b584359fc8741c0c85c4fe173d08d74e99f298c8718bcf0c6caab53434e not found: ID does not exist" containerID="4b872b584359fc8741c0c85c4fe173d08d74e99f298c8718bcf0c6caab53434e" Oct 30 09:38:30.264914 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:30.264830 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b872b584359fc8741c0c85c4fe173d08d74e99f298c8718bcf0c6caab53434e"} err="failed to get container status \"4b872b584359fc8741c0c85c4fe173d08d74e99f298c8718bcf0c6caab53434e\": rpc error: code = NotFound desc = could not find container \"4b872b584359fc8741c0c85c4fe173d08d74e99f298c8718bcf0c6caab53434e\": container with ID starting with 4b872b584359fc8741c0c85c4fe173d08d74e99f298c8718bcf0c6caab53434e not found: ID does not exist" Oct 30 09:38:30.302069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:30.302038 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8"] Oct 30 09:38:30.316255 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:30.316228 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fb1a2a/auth-test-model-fb1a2a-kserve-6748789bfc-zv5f8"] Oct 30 09:38:32.258171 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:32.258127 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6296acea-d4ae-4735-b8ab-22f8ed8d2bd8" path="/var/lib/kubelet/pods/6296acea-d4ae-4735-b8ab-22f8ed8d2bd8/volumes" Oct 30 09:38:35.651614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.644501 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc"] Oct 30 09:38:35.651614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.645529 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6296acea-d4ae-4735-b8ab-22f8ed8d2bd8" containerName="main" Oct 30 09:38:35.651614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.645550 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6296acea-d4ae-4735-b8ab-22f8ed8d2bd8" containerName="main" Oct 30 09:38:35.651614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.645578 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b4b3c5d0-f0a6-4077-b212-724b6dbe378a" containerName="istio-proxy" Oct 30 09:38:35.651614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.645609 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b3c5d0-f0a6-4077-b212-724b6dbe378a" containerName="istio-proxy" Oct 30 09:38:35.651614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.645627 5938 cpu_manager.go:414] "containerMap: removing container" podUID="64267c50-7a21-4c55-aa19-f0899841dad6" containerName="main" Oct 30 09:38:35.651614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.645634 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64267c50-7a21-4c55-aa19-f0899841dad6" containerName="main" Oct 30 09:38:35.651614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.645653 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6cb11947-f2ef-4a23-880e-17f2cccf7c0e" containerName="revoke-keys" Oct 30 09:38:35.651614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.645660 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb11947-f2ef-4a23-880e-17f2cccf7c0e" containerName="revoke-keys" Oct 30 09:38:35.651614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.645707 5938 cpu_manager.go:414] "containerMap: removing container" podUID="145760b0-8e56-4019-ad04-f0aedf4265be" containerName="revoke-keys" Oct 30 09:38:35.651614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.645714 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145760b0-8e56-4019-ad04-f0aedf4265be" containerName="revoke-keys" Oct 30 09:38:35.651614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.645872 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="145760b0-8e56-4019-ad04-f0aedf4265be" containerName="revoke-keys" Oct 30 09:38:35.651614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.645893 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6296acea-d4ae-4735-b8ab-22f8ed8d2bd8" containerName="main" Oct 30 09:38:35.651614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.645905 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4b3c5d0-f0a6-4077-b212-724b6dbe378a" containerName="istio-proxy" Oct 30 09:38:35.651614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.645913 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="64267c50-7a21-4c55-aa19-f0899841dad6" containerName="main" Oct 30 09:38:35.651614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.645922 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb11947-f2ef-4a23-880e-17f2cccf7c0e" containerName="revoke-keys" Oct 30 09:38:35.685613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.682650 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc"] Oct 30 09:38:35.685613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.682791 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" Oct 30 09:38:35.724238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.701071 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0f6f96\"" Oct 30 09:38:35.733793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.733757 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0f6f96-dockercfg-nsng5\"" Oct 30 09:38:35.771807 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.771767 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49bpd\" (UniqueName: \"kubernetes.io/projected/2089f287-3340-45e9-99d4-9e3ce1fbbe9e-kube-api-access-49bpd\") pod \"payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc\" (UID: \"2089f287-3340-45e9-99d4-9e3ce1fbbe9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" Oct 30 09:38:35.772000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.771835 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2089f287-3340-45e9-99d4-9e3ce1fbbe9e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc\" (UID: \"2089f287-3340-45e9-99d4-9e3ce1fbbe9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" Oct 30 09:38:35.872905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.872862 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49bpd\" (UniqueName: \"kubernetes.io/projected/2089f287-3340-45e9-99d4-9e3ce1fbbe9e-kube-api-access-49bpd\") pod \"payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc\" (UID: \"2089f287-3340-45e9-99d4-9e3ce1fbbe9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" Oct 30 09:38:35.873128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.872928 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2089f287-3340-45e9-99d4-9e3ce1fbbe9e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc\" (UID: \"2089f287-3340-45e9-99d4-9e3ce1fbbe9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" Oct 30 09:38:35.873697 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.873676 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2089f287-3340-45e9-99d4-9e3ce1fbbe9e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc\" (UID: \"2089f287-3340-45e9-99d4-9e3ce1fbbe9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" Oct 30 09:38:35.894432 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:35.894404 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49bpd\" (UniqueName: \"kubernetes.io/projected/2089f287-3340-45e9-99d4-9e3ce1fbbe9e-kube-api-access-49bpd\") pod \"payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc\" (UID: \"2089f287-3340-45e9-99d4-9e3ce1fbbe9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" Oct 30 09:38:36.003862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:36.003764 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" Oct 30 09:38:36.307933 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:36.307907 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc"] Oct 30 09:38:36.309691 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:38:36.309662 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2089f287_3340_45e9_99d4_9e3ce1fbbe9e.slice/crio-89db08a35dd0f7e1dfd2c159be92dd84667751a6a245e30c18e36567b8b78e65 WatchSource:0}: Error finding container 89db08a35dd0f7e1dfd2c159be92dd84667751a6a245e30c18e36567b8b78e65: Status 404 returned error can't find the container with id 89db08a35dd0f7e1dfd2c159be92dd84667751a6a245e30c18e36567b8b78e65 Oct 30 09:38:37.311571 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:37.311535 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" event={"ID":"2089f287-3340-45e9-99d4-9e3ce1fbbe9e","Type":"ContainerStarted","Data":"4cf35131c08463da4e5264099fd47802b7e80952a9fbcc0621ddc7738a5b6425"} Oct 30 09:38:37.312029 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:37.311621 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" event={"ID":"2089f287-3340-45e9-99d4-9e3ce1fbbe9e","Type":"ContainerStarted","Data":"89db08a35dd0f7e1dfd2c159be92dd84667751a6a245e30c18e36567b8b78e65"} Oct 30 09:38:37.312029 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:37.311649 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" Oct 30 09:38:37.357687 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:37.357631 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" podStartSLOduration=2.357617001 podStartE2EDuration="2.357617001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:37.354172076 +0000 UTC m=+2635.216877897" watchObservedRunningTime="2026-10-30 09:38:37.357617001 +0000 UTC m=+2635.220322818" Oct 30 09:38:39.900266 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:39.900226 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" podUID="4e15c89b-93ef-4515-a947-930e43925d64" containerName="main" containerID="cri-o://cd87fd74f6ab2155584bf6914e599a0667a64874c41cd75a3b98ce3a4868ea3e" gracePeriod=45 Oct 30 09:38:40.192213 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.192190 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:38:40.234647 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.234620 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-model-cache\") pod \"4e15c89b-93ef-4515-a947-930e43925d64\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " Oct 30 09:38:40.234827 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.234674 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e15c89b-93ef-4515-a947-930e43925d64-kube-api-access-8qbfv\" (UniqueName: \"kubernetes.io/projected/4e15c89b-93ef-4515-a947-930e43925d64-kube-api-access-8qbfv\") pod \"4e15c89b-93ef-4515-a947-930e43925d64\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " Oct 30 09:38:40.234827 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.234802 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-dshm\") pod \"4e15c89b-93ef-4515-a947-930e43925d64\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " Oct 30 09:38:40.234959 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.234885 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e15c89b-93ef-4515-a947-930e43925d64-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e15c89b-93ef-4515-a947-930e43925d64-tls-certs\") pod \"4e15c89b-93ef-4515-a947-930e43925d64\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " Oct 30 09:38:40.234959 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.234930 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-home\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-home\") pod \"4e15c89b-93ef-4515-a947-930e43925d64\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " Oct 30 09:38:40.234959 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.234926 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-model-cache" pod "4e15c89b-93ef-4515-a947-930e43925d64" (UID: "4e15c89b-93ef-4515-a947-930e43925d64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:40.235145 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.234962 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-tmp-dir\") pod \"4e15c89b-93ef-4515-a947-930e43925d64\" (UID: \"4e15c89b-93ef-4515-a947-930e43925d64\") " Oct 30 09:38:40.237339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.235470 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-home" pod "4e15c89b-93ef-4515-a947-930e43925d64" (UID: "4e15c89b-93ef-4515-a947-930e43925d64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:40.237339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.235661 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:40.237339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.235681 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:40.237339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.235656 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-tmp-dir" pod "4e15c89b-93ef-4515-a947-930e43925d64" (UID: "4e15c89b-93ef-4515-a947-930e43925d64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:40.238553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.238187 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-dshm" pod "4e15c89b-93ef-4515-a947-930e43925d64" (UID: "4e15c89b-93ef-4515-a947-930e43925d64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:40.238553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.238257 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e15c89b-93ef-4515-a947-930e43925d64-tls-certs" pod "4e15c89b-93ef-4515-a947-930e43925d64" (UID: "4e15c89b-93ef-4515-a947-930e43925d64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:38:40.238553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.238259 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e15c89b-93ef-4515-a947-930e43925d64-kube-api-access-8qbfv" pod "4e15c89b-93ef-4515-a947-930e43925d64" (UID: "4e15c89b-93ef-4515-a947-930e43925d64"). InnerVolumeSpecName "kube-api-access-8qbfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:40.334342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.334313 5938 generic.go:356] "Generic (PLEG): container finished" podID="4e15c89b-93ef-4515-a947-930e43925d64" containerID="cd87fd74f6ab2155584bf6914e599a0667a64874c41cd75a3b98ce3a4868ea3e" exitCode=0 Oct 30 09:38:40.334489 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.334364 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" event={"ID":"4e15c89b-93ef-4515-a947-930e43925d64","Type":"ContainerDied","Data":"cd87fd74f6ab2155584bf6914e599a0667a64874c41cd75a3b98ce3a4868ea3e"} Oct 30 09:38:40.334489 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.334372 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" Oct 30 09:38:40.334489 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.334393 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw" event={"ID":"4e15c89b-93ef-4515-a947-930e43925d64","Type":"ContainerDied","Data":"c349f7b24be69e312e768d53128d1285ec5affcfe2bd35c89910ac9efff02cc9"} Oct 30 09:38:40.334489 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.334409 5938 scope.go:122] "RemoveContainer" containerID="cd87fd74f6ab2155584bf6914e599a0667a64874c41cd75a3b98ce3a4868ea3e" Oct 30 09:38:40.337319 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.337294 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qbfv\" (UniqueName: \"kubernetes.io/projected/4e15c89b-93ef-4515-a947-930e43925d64-kube-api-access-8qbfv\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:40.337461 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.337325 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:40.337461 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.337342 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e15c89b-93ef-4515-a947-930e43925d64-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:40.337461 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.337357 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e15c89b-93ef-4515-a947-930e43925d64-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:40.345643 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.345628 5938 scope.go:122] "RemoveContainer" containerID="cd87fd74f6ab2155584bf6914e599a0667a64874c41cd75a3b98ce3a4868ea3e" Oct 30 09:38:40.345929 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:38:40.345911 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd87fd74f6ab2155584bf6914e599a0667a64874c41cd75a3b98ce3a4868ea3e\": container with ID starting with cd87fd74f6ab2155584bf6914e599a0667a64874c41cd75a3b98ce3a4868ea3e not found: ID does not exist" containerID="cd87fd74f6ab2155584bf6914e599a0667a64874c41cd75a3b98ce3a4868ea3e" Oct 30 09:38:40.345973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.345937 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd87fd74f6ab2155584bf6914e599a0667a64874c41cd75a3b98ce3a4868ea3e"} err="failed to get container status \"cd87fd74f6ab2155584bf6914e599a0667a64874c41cd75a3b98ce3a4868ea3e\": rpc error: code = NotFound desc = could not find container \"cd87fd74f6ab2155584bf6914e599a0667a64874c41cd75a3b98ce3a4868ea3e\": container with ID starting with cd87fd74f6ab2155584bf6914e599a0667a64874c41cd75a3b98ce3a4868ea3e not found: ID does not exist" Oct 30 09:38:40.405915 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.405893 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw"] Oct 30 09:38:40.419378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:40.419329 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-0dc148/auth-test-model-0dc148-kserve-5c9f84fb7c-8hntw"] Oct 30 09:38:42.258512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:42.258479 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e15c89b-93ef-4515-a947-930e43925d64" path="/var/lib/kubelet/pods/4e15c89b-93ef-4515-a947-930e43925d64/volumes" Oct 30 09:38:43.488274 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:43.488243 5938 scope.go:122] "RemoveContainer" containerID="bc8438225611422621417b01e4835794948c2095448a7b413ae0da25b3f8a910" Oct 30 09:38:43.501811 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:43.501787 5938 scope.go:122] "RemoveContainer" containerID="ce50d9d5a6ab7e0c739e1b42e6649fadc9bc738f4fccd56fe39649aad5674cc3" Oct 30 09:38:43.511923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:43.511900 5938 scope.go:122] "RemoveContainer" containerID="5a5c30b15822b2444aaa795dfe5e042f7be3b7588055b0917f2a4de15df4114f" Oct 30 09:38:43.521072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:43.521049 5938 scope.go:122] "RemoveContainer" containerID="b15d5261143893850b67704e268c190af929eff9e2af163438325eea10cf1800" Oct 30 09:38:43.530787 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:43.530770 5938 scope.go:122] "RemoveContainer" containerID="d7d5b1a093fcfa6c669c88137077a83d1bbe5ab29156091e0f28a457921cc1e4" Oct 30 09:38:45.809930 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:38:45.809895 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:38:47.246547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:47.246510 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11387-146e92"] Oct 30 09:38:47.247466 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:47.247441 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4e15c89b-93ef-4515-a947-930e43925d64" containerName="main" Oct 30 09:38:47.247466 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:47.247466 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e15c89b-93ef-4515-a947-930e43925d64" containerName="main" Oct 30 09:38:47.247724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:47.247708 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e15c89b-93ef-4515-a947-930e43925d64" containerName="main" Oct 30 09:38:47.253344 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:47.253316 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11387-146e92" Oct 30 09:38:47.265520 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:47.265493 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11387-146e92"] Oct 30 09:38:47.275320 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:47.275297 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jjt9f\"" Oct 30 09:38:47.337053 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:47.337001 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz62f\" (UniqueName: \"kubernetes.io/projected/89f1bece-83b5-4f27-9da9-53887c9c8e3e-kube-api-access-nz62f\") pod \"test-curl-11387-146e92\" (UID: \"89f1bece-83b5-4f27-9da9-53887c9c8e3e\") " pod="opendatahub/test-curl-11387-146e92" Oct 30 09:38:47.438680 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:47.438634 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz62f\" (UniqueName: \"kubernetes.io/projected/89f1bece-83b5-4f27-9da9-53887c9c8e3e-kube-api-access-nz62f\") pod \"test-curl-11387-146e92\" (UID: \"89f1bece-83b5-4f27-9da9-53887c9c8e3e\") " pod="opendatahub/test-curl-11387-146e92" Oct 30 09:38:47.459354 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:47.459313 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz62f\" (UniqueName: \"kubernetes.io/projected/89f1bece-83b5-4f27-9da9-53887c9c8e3e-kube-api-access-nz62f\") pod \"test-curl-11387-146e92\" (UID: \"89f1bece-83b5-4f27-9da9-53887c9c8e3e\") " pod="opendatahub/test-curl-11387-146e92" Oct 30 09:38:47.567318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:47.567240 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11387-146e92" Oct 30 09:38:47.868886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:47.868859 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11387-146e92"] Oct 30 09:38:47.869863 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:38:47.869826 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89f1bece_83b5_4f27_9da9_53887c9c8e3e.slice/crio-6230c034337de475f8d2e44dd03a248f1bb8910c2e751ec79c8d4eaaca233853 WatchSource:0}: Error finding container 6230c034337de475f8d2e44dd03a248f1bb8910c2e751ec79c8d4eaaca233853: Status 404 returned error can't find the container with id 6230c034337de475f8d2e44dd03a248f1bb8910c2e751ec79c8d4eaaca233853 Oct 30 09:38:47.871878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:47.871861 5938 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:38:48.322514 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:48.322430 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" Oct 30 09:38:48.401141 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:48.401107 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11387-146e92" event={"ID":"89f1bece-83b5-4f27-9da9-53887c9c8e3e","Type":"ContainerStarted","Data":"6230c034337de475f8d2e44dd03a248f1bb8910c2e751ec79c8d4eaaca233853"} Oct 30 09:38:49.410577 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:49.410527 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11387-146e92" event={"ID":"89f1bece-83b5-4f27-9da9-53887c9c8e3e","Type":"ContainerStarted","Data":"03c188ce7d09ceb4355eb3938016440ad4deda1b502cc32378070abbb797da73"} Oct 30 09:38:49.455328 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:49.455271 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11387-146e92" podStartSLOduration=1.8576845 podStartE2EDuration="2.455255164s" totalImagesPullingTime="597.570664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:49.452540004 +0000 UTC m=+2647.315245826" watchObservedRunningTime="2026-10-30 09:38:49.455255164 +0000 UTC m=+2647.317961048" Oct 30 09:38:50.346617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.346234 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11387-146e92"] Oct 30 09:38:50.356954 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.356919 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11387-146e92"] Oct 30 09:38:50.571405 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.571376 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t"] Oct 30 09:38:50.572147 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.572130 5938 cpu_manager.go:414] "containerMap: removing container" podUID="89f1bece-83b5-4f27-9da9-53887c9c8e3e" containerName="test-curl-11387-146e92" Oct 30 09:38:50.572198 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.572149 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f1bece-83b5-4f27-9da9-53887c9c8e3e" containerName="test-curl-11387-146e92" Oct 30 09:38:50.572260 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.572251 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f1bece-83b5-4f27-9da9-53887c9c8e3e" containerName="test-curl-11387-146e92" Oct 30 09:38:50.575969 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.575952 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t" Oct 30 09:38:50.595858 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.595823 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t"] Oct 30 09:38:50.605501 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.605424 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:38:50.606471 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.606416 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:38:50.697318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.697271 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d598e56-h9r9t\" (UID: \"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t" Oct 30 09:38:50.697520 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.697331 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d598e56-h9r9t\" (UID: \"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t" Oct 30 09:38:50.697520 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.697419 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skzb4\" (UniqueName: \"kubernetes.io/projected/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-kube-api-access-skzb4\") pod \"maas-api-revoke-sub-6d598e56-h9r9t\" (UID: \"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t" Oct 30 09:38:50.798386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.798347 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d598e56-h9r9t\" (UID: \"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t" Oct 30 09:38:50.798386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.798399 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d598e56-h9r9t\" (UID: \"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t" Oct 30 09:38:50.798693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.798473 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skzb4\" (UniqueName: \"kubernetes.io/projected/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-kube-api-access-skzb4\") pod \"maas-api-revoke-sub-6d598e56-h9r9t\" (UID: \"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t" Oct 30 09:38:50.799177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.799153 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d598e56-h9r9t\" (UID: \"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t" Oct 30 09:38:50.801154 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.801134 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d598e56-h9r9t\" (UID: \"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t" Oct 30 09:38:50.818991 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.818964 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skzb4\" (UniqueName: \"kubernetes.io/projected/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-kube-api-access-skzb4\") pod \"maas-api-revoke-sub-6d598e56-h9r9t\" (UID: \"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t" Oct 30 09:38:50.889806 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:50.889779 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t" Oct 30 09:38:51.211471 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:38:51.211186 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e34684_56f1_4eae_ba7d_0b7abbdcff8e.slice/crio-5baa57a62f557a174af87ee8edabe86ebf5bca3170f39a442a31a20ecc1850d0 WatchSource:0}: Error finding container 5baa57a62f557a174af87ee8edabe86ebf5bca3170f39a442a31a20ecc1850d0: Status 404 returned error can't find the container with id 5baa57a62f557a174af87ee8edabe86ebf5bca3170f39a442a31a20ecc1850d0 Oct 30 09:38:51.213927 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:51.213901 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t"] Oct 30 09:38:51.436566 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:51.436496 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t" event={"ID":"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e","Type":"ContainerStarted","Data":"5baa57a62f557a174af87ee8edabe86ebf5bca3170f39a442a31a20ecc1850d0"} Oct 30 09:38:51.436793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:51.436704 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11387-146e92" podUID="89f1bece-83b5-4f27-9da9-53887c9c8e3e" containerName="test-curl-11387-146e92" containerID="cri-o://03c188ce7d09ceb4355eb3938016440ad4deda1b502cc32378070abbb797da73" gracePeriod=2 Oct 30 09:38:51.451630 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:38:51.451573 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11387-146e92\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="89f1bece-83b5-4f27-9da9-53887c9c8e3e" pod="opendatahub/test-curl-11387-146e92" Oct 30 09:38:52.270372 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:38:52.270333 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11387-146e92\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="89f1bece-83b5-4f27-9da9-53887c9c8e3e" pod="opendatahub/test-curl-11387-146e92" Oct 30 09:38:52.445028 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:52.444935 5938 generic.go:356] "Generic (PLEG): container finished" podID="a3e34684-56f1-4eae-ba7d-0b7abbdcff8e" containerID="e8069d7e140878ba5fd9617f6774d40aa07dbf68dfd737b0c7560183d583fdb9" exitCode=0 Oct 30 09:38:52.445213 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:52.445021 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t" event={"ID":"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e","Type":"ContainerDied","Data":"e8069d7e140878ba5fd9617f6774d40aa07dbf68dfd737b0c7560183d583fdb9"} Oct 30 09:38:53.632829 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:53.632802 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t" Oct 30 09:38:53.684777 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:53.684749 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11387-146e92" Oct 30 09:38:53.745568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:53.745476 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-maas-api-openshift-ca\") pod \"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e\" (UID: \"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e\") " Oct 30 09:38:53.745781 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:53.745647 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-maas-api-xks-ca\") pod \"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e\" (UID: \"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e\") " Oct 30 09:38:53.745781 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:53.745684 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f1bece-83b5-4f27-9da9-53887c9c8e3e-kube-api-access-nz62f\" (UniqueName: \"kubernetes.io/projected/89f1bece-83b5-4f27-9da9-53887c9c8e3e-kube-api-access-nz62f\") pod \"89f1bece-83b5-4f27-9da9-53887c9c8e3e\" (UID: \"89f1bece-83b5-4f27-9da9-53887c9c8e3e\") " Oct 30 09:38:53.745921 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:53.745783 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-kube-api-access-skzb4\" (UniqueName: \"kubernetes.io/projected/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-kube-api-access-skzb4\") pod \"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e\" (UID: \"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e\") " Oct 30 09:38:53.745985 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:53.745907 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-maas-api-openshift-ca" pod "a3e34684-56f1-4eae-ba7d-0b7abbdcff8e" (UID: "a3e34684-56f1-4eae-ba7d-0b7abbdcff8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:38:53.746248 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:53.746229 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:53.747967 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:53.747941 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-maas-api-xks-ca" pod "a3e34684-56f1-4eae-ba7d-0b7abbdcff8e" (UID: "a3e34684-56f1-4eae-ba7d-0b7abbdcff8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:38:53.748081 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:53.748046 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-kube-api-access-skzb4" pod "a3e34684-56f1-4eae-ba7d-0b7abbdcff8e" (UID: "a3e34684-56f1-4eae-ba7d-0b7abbdcff8e"). InnerVolumeSpecName "kube-api-access-skzb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:53.748129 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:53.748085 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f1bece-83b5-4f27-9da9-53887c9c8e3e-kube-api-access-nz62f" pod "89f1bece-83b5-4f27-9da9-53887c9c8e3e" (UID: "89f1bece-83b5-4f27-9da9-53887c9c8e3e"). InnerVolumeSpecName "kube-api-access-nz62f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:53.847849 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:53.847812 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:53.847849 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:53.847841 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz62f\" (UniqueName: \"kubernetes.io/projected/89f1bece-83b5-4f27-9da9-53887c9c8e3e-kube-api-access-nz62f\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:53.847849 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:53.847856 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skzb4\" (UniqueName: \"kubernetes.io/projected/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e-kube-api-access-skzb4\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:38:54.258434 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:54.258402 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89f1bece-83b5-4f27-9da9-53887c9c8e3e" path="/var/lib/kubelet/pods/89f1bece-83b5-4f27-9da9-53887c9c8e3e/volumes" Oct 30 09:38:54.462514 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:54.462489 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t" Oct 30 09:38:54.462683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:54.462527 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t" event={"ID":"a3e34684-56f1-4eae-ba7d-0b7abbdcff8e","Type":"ContainerDied","Data":"5baa57a62f557a174af87ee8edabe86ebf5bca3170f39a442a31a20ecc1850d0"} Oct 30 09:38:54.462683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:54.462581 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5baa57a62f557a174af87ee8edabe86ebf5bca3170f39a442a31a20ecc1850d0" Oct 30 09:38:54.463861 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:54.463837 5938 generic.go:356] "Generic (PLEG): container finished" podID="89f1bece-83b5-4f27-9da9-53887c9c8e3e" containerID="03c188ce7d09ceb4355eb3938016440ad4deda1b502cc32378070abbb797da73" exitCode=137 Oct 30 09:38:54.464006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:54.463873 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11387-146e92" Oct 30 09:38:54.464006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:54.463935 5938 scope.go:122] "RemoveContainer" containerID="03c188ce7d09ceb4355eb3938016440ad4deda1b502cc32378070abbb797da73" Oct 30 09:38:54.473955 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:54.473936 5938 scope.go:122] "RemoveContainer" containerID="03c188ce7d09ceb4355eb3938016440ad4deda1b502cc32378070abbb797da73" Oct 30 09:38:54.474232 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:38:54.474208 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c188ce7d09ceb4355eb3938016440ad4deda1b502cc32378070abbb797da73\": container with ID starting with 03c188ce7d09ceb4355eb3938016440ad4deda1b502cc32378070abbb797da73 not found: ID does not exist" containerID="03c188ce7d09ceb4355eb3938016440ad4deda1b502cc32378070abbb797da73" Oct 30 09:38:54.474281 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:38:54.474240 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c188ce7d09ceb4355eb3938016440ad4deda1b502cc32378070abbb797da73"} err="failed to get container status \"03c188ce7d09ceb4355eb3938016440ad4deda1b502cc32378070abbb797da73\": rpc error: code = NotFound desc = could not find container \"03c188ce7d09ceb4355eb3938016440ad4deda1b502cc32378070abbb797da73\": container with ID starting with 03c188ce7d09ceb4355eb3938016440ad4deda1b502cc32378070abbb797da73 not found: ID does not exist" Oct 30 09:39:00.757519 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.757479 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl"] Oct 30 09:39:00.758210 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.758193 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a3e34684-56f1-4eae-ba7d-0b7abbdcff8e" containerName="revoke-keys" Oct 30 09:39:00.758210 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.758210 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e34684-56f1-4eae-ba7d-0b7abbdcff8e" containerName="revoke-keys" Oct 30 09:39:00.758339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.758318 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e34684-56f1-4eae-ba7d-0b7abbdcff8e" containerName="revoke-keys" Oct 30 09:39:00.764016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.763998 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:39:00.781947 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.781920 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl"] Oct 30 09:39:00.791378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.791351 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d60c68-serving-cert\"" Oct 30 09:39:00.844449 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.844415 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ee9234-06c5-400c-b8b5-575e5ed17516-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl\" (UID: \"47ee9234-06c5-400c-b8b5-575e5ed17516\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:39:00.844667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.844493 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ee9234-06c5-400c-b8b5-575e5ed17516-metrics-certs\") pod \"maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl\" (UID: \"47ee9234-06c5-400c-b8b5-575e5ed17516\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:39:00.844667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.844513 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ee9234-06c5-400c-b8b5-575e5ed17516-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl\" (UID: \"47ee9234-06c5-400c-b8b5-575e5ed17516\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:39:00.844667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.844535 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw8kw\" (UniqueName: \"kubernetes.io/projected/47ee9234-06c5-400c-b8b5-575e5ed17516-kube-api-access-sw8kw\") pod \"maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl\" (UID: \"47ee9234-06c5-400c-b8b5-575e5ed17516\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:39:00.946139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.946098 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ee9234-06c5-400c-b8b5-575e5ed17516-metrics-certs\") pod \"maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl\" (UID: \"47ee9234-06c5-400c-b8b5-575e5ed17516\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:39:00.946139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.946141 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ee9234-06c5-400c-b8b5-575e5ed17516-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl\" (UID: \"47ee9234-06c5-400c-b8b5-575e5ed17516\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:39:00.946363 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.946165 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw8kw\" (UniqueName: \"kubernetes.io/projected/47ee9234-06c5-400c-b8b5-575e5ed17516-kube-api-access-sw8kw\") pod \"maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl\" (UID: \"47ee9234-06c5-400c-b8b5-575e5ed17516\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:39:00.946363 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.946356 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ee9234-06c5-400c-b8b5-575e5ed17516-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl\" (UID: \"47ee9234-06c5-400c-b8b5-575e5ed17516\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:39:00.947072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.947048 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ee9234-06c5-400c-b8b5-575e5ed17516-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl\" (UID: \"47ee9234-06c5-400c-b8b5-575e5ed17516\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:39:00.948953 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.948926 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ee9234-06c5-400c-b8b5-575e5ed17516-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl\" (UID: \"47ee9234-06c5-400c-b8b5-575e5ed17516\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:39:00.948953 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.948948 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ee9234-06c5-400c-b8b5-575e5ed17516-metrics-certs\") pod \"maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl\" (UID: \"47ee9234-06c5-400c-b8b5-575e5ed17516\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:39:00.969043 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:00.969016 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw8kw\" (UniqueName: \"kubernetes.io/projected/47ee9234-06c5-400c-b8b5-575e5ed17516-kube-api-access-sw8kw\") pod \"maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl\" (UID: \"47ee9234-06c5-400c-b8b5-575e5ed17516\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:39:01.022244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.022158 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct"] Oct 30 09:39:01.032118 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.032082 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" Oct 30 09:39:01.045481 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.045447 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct"] Oct 30 09:39:01.047992 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.047965 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:39:01.048222 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.048203 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:39:01.078801 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.078765 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:39:01.157986 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.150069 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c3c0b4-78kct\" (UID: \"3d0eda9a-f78d-4624-8514-0b4f2a1531e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" Oct 30 09:39:01.157986 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.150141 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptzbp\" (UniqueName: \"kubernetes.io/projected/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-kube-api-access-ptzbp\") pod \"maas-api-revoke-sub-80c3c0b4-78kct\" (UID: \"3d0eda9a-f78d-4624-8514-0b4f2a1531e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" Oct 30 09:39:01.157986 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.150351 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c3c0b4-78kct\" (UID: \"3d0eda9a-f78d-4624-8514-0b4f2a1531e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" Oct 30 09:39:01.251623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.251569 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c3c0b4-78kct\" (UID: \"3d0eda9a-f78d-4624-8514-0b4f2a1531e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" Oct 30 09:39:01.251795 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.251651 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptzbp\" (UniqueName: \"kubernetes.io/projected/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-kube-api-access-ptzbp\") pod \"maas-api-revoke-sub-80c3c0b4-78kct\" (UID: \"3d0eda9a-f78d-4624-8514-0b4f2a1531e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" Oct 30 09:39:01.251795 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.251775 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c3c0b4-78kct\" (UID: \"3d0eda9a-f78d-4624-8514-0b4f2a1531e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" Oct 30 09:39:01.252471 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.252397 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c3c0b4-78kct\" (UID: \"3d0eda9a-f78d-4624-8514-0b4f2a1531e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" Oct 30 09:39:01.254209 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.254186 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c3c0b4-78kct\" (UID: \"3d0eda9a-f78d-4624-8514-0b4f2a1531e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" Oct 30 09:39:01.272387 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.272313 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptzbp\" (UniqueName: \"kubernetes.io/projected/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-kube-api-access-ptzbp\") pod \"maas-api-revoke-sub-80c3c0b4-78kct\" (UID: \"3d0eda9a-f78d-4624-8514-0b4f2a1531e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" Oct 30 09:39:01.349485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.349448 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" Oct 30 09:39:01.387727 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.387703 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl"] Oct 30 09:39:01.388915 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:39:01.388887 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ee9234_06c5_400c_b8b5_575e5ed17516.slice/crio-1e133f4bdb772ca384cc4b0d3e3cfbe85f5bb794651a49fc415cf717ac385d76 WatchSource:0}: Error finding container 1e133f4bdb772ca384cc4b0d3e3cfbe85f5bb794651a49fc415cf717ac385d76: Status 404 returned error can't find the container with id 1e133f4bdb772ca384cc4b0d3e3cfbe85f5bb794651a49fc415cf717ac385d76 Oct 30 09:39:01.543670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.543166 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" event={"ID":"47ee9234-06c5-400c-b8b5-575e5ed17516","Type":"ContainerStarted","Data":"1e133f4bdb772ca384cc4b0d3e3cfbe85f5bb794651a49fc415cf717ac385d76"} Oct 30 09:39:01.661961 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:01.661935 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct"] Oct 30 09:39:01.662514 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:39:01.662493 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0eda9a_f78d_4624_8514_0b4f2a1531e8.slice/crio-1638ef1112241c6ccfbdc4c7ea426922a40e3dc94312442191e2c1052de01205 WatchSource:0}: Error finding container 1638ef1112241c6ccfbdc4c7ea426922a40e3dc94312442191e2c1052de01205: Status 404 returned error can't find the container with id 1638ef1112241c6ccfbdc4c7ea426922a40e3dc94312442191e2c1052de01205 Oct 30 09:39:02.562443 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:02.562400 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" event={"ID":"47ee9234-06c5-400c-b8b5-575e5ed17516","Type":"ContainerStarted","Data":"54300a36659e6a3e61b628822738a2e69b34abb54ce1ceef255f95cce23cf8de"} Oct 30 09:39:02.567542 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:02.565054 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:39:02.572557 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:02.572524 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" event={"ID":"3d0eda9a-f78d-4624-8514-0b4f2a1531e8","Type":"ContainerStarted","Data":"ebc4f305a4b8128c125b5759ea5c31725a719a25de9d09b0c3c927e744d11cf8"} Oct 30 09:39:02.572731 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:02.572579 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" event={"ID":"3d0eda9a-f78d-4624-8514-0b4f2a1531e8","Type":"ContainerStarted","Data":"1638ef1112241c6ccfbdc4c7ea426922a40e3dc94312442191e2c1052de01205"} Oct 30 09:39:02.614380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:02.614330 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" podStartSLOduration=2.2718019910000002 podStartE2EDuration="2.614314265s" totalImagesPullingTime="342.512274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:02.612697781 +0000 UTC m=+2660.475403603" watchObservedRunningTime="2026-10-30 09:39:02.614314265 +0000 UTC m=+2660.477020085" Oct 30 09:39:02.652888 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:02.652823 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" podStartSLOduration=2.6528026000000002 podStartE2EDuration="2.6528026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:02.652578372 +0000 UTC m=+2660.515284191" watchObservedRunningTime="2026-10-30 09:39:02.6528026 +0000 UTC m=+2660.515508419" Oct 30 09:39:03.580977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:03.580942 5938 generic.go:356] "Generic (PLEG): container finished" podID="3d0eda9a-f78d-4624-8514-0b4f2a1531e8" containerID="ebc4f305a4b8128c125b5759ea5c31725a719a25de9d09b0c3c927e744d11cf8" exitCode=0 Oct 30 09:39:03.581486 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:03.580986 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" event={"ID":"3d0eda9a-f78d-4624-8514-0b4f2a1531e8","Type":"ContainerDied","Data":"ebc4f305a4b8128c125b5759ea5c31725a719a25de9d09b0c3c927e744d11cf8"} Oct 30 09:39:03.936821 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:03.936787 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg"] Oct 30 09:39:03.940849 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:03.940831 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:39:03.957847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:03.957824 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-0f6f96-serving-cert\"" Oct 30 09:39:03.962703 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:03.962673 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg"] Oct 30 09:39:04.097496 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.097467 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc7bd19a-03fc-4246-b529-dd67becce442-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg\" (UID: \"fc7bd19a-03fc-4246-b529-dd67becce442\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:39:04.097722 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.097524 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7bd19a-03fc-4246-b529-dd67becce442-metrics-certs\") pod \"maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg\" (UID: \"fc7bd19a-03fc-4246-b529-dd67becce442\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:39:04.097877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.097853 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5659\" (UniqueName: \"kubernetes.io/projected/fc7bd19a-03fc-4246-b529-dd67becce442-kube-api-access-x5659\") pod \"maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg\" (UID: \"fc7bd19a-03fc-4246-b529-dd67becce442\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:39:04.098019 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.097969 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc7bd19a-03fc-4246-b529-dd67becce442-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg\" (UID: \"fc7bd19a-03fc-4246-b529-dd67becce442\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:39:04.198943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.198849 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5659\" (UniqueName: \"kubernetes.io/projected/fc7bd19a-03fc-4246-b529-dd67becce442-kube-api-access-x5659\") pod \"maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg\" (UID: \"fc7bd19a-03fc-4246-b529-dd67becce442\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:39:04.198943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.198917 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc7bd19a-03fc-4246-b529-dd67becce442-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg\" (UID: \"fc7bd19a-03fc-4246-b529-dd67becce442\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:39:04.198943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.198943 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc7bd19a-03fc-4246-b529-dd67becce442-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg\" (UID: \"fc7bd19a-03fc-4246-b529-dd67becce442\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:39:04.199264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.198983 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7bd19a-03fc-4246-b529-dd67becce442-metrics-certs\") pod \"maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg\" (UID: \"fc7bd19a-03fc-4246-b529-dd67becce442\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:39:04.199910 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.199814 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc7bd19a-03fc-4246-b529-dd67becce442-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg\" (UID: \"fc7bd19a-03fc-4246-b529-dd67becce442\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:39:04.201850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.201831 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc7bd19a-03fc-4246-b529-dd67becce442-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg\" (UID: \"fc7bd19a-03fc-4246-b529-dd67becce442\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:39:04.201933 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.201882 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7bd19a-03fc-4246-b529-dd67becce442-metrics-certs\") pod \"maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg\" (UID: \"fc7bd19a-03fc-4246-b529-dd67becce442\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:39:04.221937 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.221898 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5659\" (UniqueName: \"kubernetes.io/projected/fc7bd19a-03fc-4246-b529-dd67becce442-kube-api-access-x5659\") pod \"maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg\" (UID: \"fc7bd19a-03fc-4246-b529-dd67becce442\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:39:04.254822 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.254786 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:39:04.561087 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.561062 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg"] Oct 30 09:39:04.562376 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:39:04.562349 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7bd19a_03fc_4246_b529_dd67becce442.slice/crio-c762f2d0d80b31d3b712e4fbb6f1f620bba340681f66f20f3443727192a93819 WatchSource:0}: Error finding container c762f2d0d80b31d3b712e4fbb6f1f620bba340681f66f20f3443727192a93819: Status 404 returned error can't find the container with id c762f2d0d80b31d3b712e4fbb6f1f620bba340681f66f20f3443727192a93819 Oct 30 09:39:04.590401 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.590366 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" event={"ID":"fc7bd19a-03fc-4246-b529-dd67becce442","Type":"ContainerStarted","Data":"c762f2d0d80b31d3b712e4fbb6f1f620bba340681f66f20f3443727192a93819"} Oct 30 09:39:04.743328 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.743283 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" Oct 30 09:39:04.910495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.910463 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-maas-api-openshift-ca\") pod \"3d0eda9a-f78d-4624-8514-0b4f2a1531e8\" (UID: \"3d0eda9a-f78d-4624-8514-0b4f2a1531e8\") " Oct 30 09:39:04.911546 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.910728 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-kube-api-access-ptzbp\" (UniqueName: \"kubernetes.io/projected/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-kube-api-access-ptzbp\") pod \"3d0eda9a-f78d-4624-8514-0b4f2a1531e8\" (UID: \"3d0eda9a-f78d-4624-8514-0b4f2a1531e8\") " Oct 30 09:39:04.911546 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.910773 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-maas-api-xks-ca\") pod \"3d0eda9a-f78d-4624-8514-0b4f2a1531e8\" (UID: \"3d0eda9a-f78d-4624-8514-0b4f2a1531e8\") " Oct 30 09:39:04.911784 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.911702 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-maas-api-openshift-ca" pod "3d0eda9a-f78d-4624-8514-0b4f2a1531e8" (UID: "3d0eda9a-f78d-4624-8514-0b4f2a1531e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:39:04.915777 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.914289 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-maas-api-xks-ca" pod "3d0eda9a-f78d-4624-8514-0b4f2a1531e8" (UID: "3d0eda9a-f78d-4624-8514-0b4f2a1531e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:39:04.915777 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:04.914412 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-kube-api-access-ptzbp" pod "3d0eda9a-f78d-4624-8514-0b4f2a1531e8" (UID: "3d0eda9a-f78d-4624-8514-0b4f2a1531e8"). InnerVolumeSpecName "kube-api-access-ptzbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:39:05.063279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:05.063135 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptzbp\" (UniqueName: \"kubernetes.io/projected/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-kube-api-access-ptzbp\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:05.063279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:05.063172 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:05.063279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:05.063188 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d0eda9a-f78d-4624-8514-0b4f2a1531e8-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:05.600166 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:05.600069 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" event={"ID":"fc7bd19a-03fc-4246-b529-dd67becce442","Type":"ContainerStarted","Data":"63957b9fa187491c03319456f5862417dbfba0b0bb8c569b08e8c795c03f44d6"} Oct 30 09:39:05.600665 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:05.600174 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:39:05.601734 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:05.601714 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" Oct 30 09:39:05.601856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:05.601741 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct" event={"ID":"3d0eda9a-f78d-4624-8514-0b4f2a1531e8","Type":"ContainerDied","Data":"1638ef1112241c6ccfbdc4c7ea426922a40e3dc94312442191e2c1052de01205"} Oct 30 09:39:05.601856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:05.601779 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1638ef1112241c6ccfbdc4c7ea426922a40e3dc94312442191e2c1052de01205" Oct 30 09:39:05.645913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:05.645868 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" podStartSLOduration=2.290718164 podStartE2EDuration="2.64585313s" totalImagesPullingTime="355.134966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:05.643498976 +0000 UTC m=+2663.506204797" watchObservedRunningTime="2026-10-30 09:39:05.64585313 +0000 UTC m=+2663.508558951" Oct 30 09:39:08.587154 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:08.587126 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:39:11.615923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.615893 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:39:11.676923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.676889 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks"] Oct 30 09:39:11.677734 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.677715 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3d0eda9a-f78d-4624-8514-0b4f2a1531e8" containerName="revoke-keys" Oct 30 09:39:11.677814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.677735 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d0eda9a-f78d-4624-8514-0b4f2a1531e8" containerName="revoke-keys" Oct 30 09:39:11.677876 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.677852 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d0eda9a-f78d-4624-8514-0b4f2a1531e8" containerName="revoke-keys" Oct 30 09:39:11.681660 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.681635 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" Oct 30 09:39:11.698199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.698159 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a525bd-dockercfg-pp9bb\"" Oct 30 09:39:11.703816 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.703788 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks"] Oct 30 09:39:11.704294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.704211 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a525bd\"" Oct 30 09:39:11.767173 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.767140 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a880792-dae1-47f4-96e9-7df47ce563cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks\" (UID: \"7a880792-dae1-47f4-96e9-7df47ce563cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" Oct 30 09:39:11.767364 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.767194 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2254\" (UniqueName: \"kubernetes.io/projected/7a880792-dae1-47f4-96e9-7df47ce563cf-kube-api-access-k2254\") pod \"payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks\" (UID: \"7a880792-dae1-47f4-96e9-7df47ce563cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" Oct 30 09:39:11.777525 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.777494 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh"] Oct 30 09:39:11.781854 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.781835 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" Oct 30 09:39:11.815618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.808685 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh"] Oct 30 09:39:11.880670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.879799 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a880792-dae1-47f4-96e9-7df47ce563cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks\" (UID: \"7a880792-dae1-47f4-96e9-7df47ce563cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" Oct 30 09:39:11.880670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.879879 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2254\" (UniqueName: \"kubernetes.io/projected/7a880792-dae1-47f4-96e9-7df47ce563cf-kube-api-access-k2254\") pod \"payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks\" (UID: \"7a880792-dae1-47f4-96e9-7df47ce563cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" Oct 30 09:39:11.899641 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.881949 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a880792-dae1-47f4-96e9-7df47ce563cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks\" (UID: \"7a880792-dae1-47f4-96e9-7df47ce563cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" Oct 30 09:39:11.908191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.908159 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2254\" (UniqueName: \"kubernetes.io/projected/7a880792-dae1-47f4-96e9-7df47ce563cf-kube-api-access-k2254\") pod \"payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks\" (UID: \"7a880792-dae1-47f4-96e9-7df47ce563cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" Oct 30 09:39:11.981814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.981785 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbmb8\" (UniqueName: \"kubernetes.io/projected/75d0141a-4aa4-41fe-b47d-1e564cc52302-kube-api-access-dbmb8\") pod \"payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh\" (UID: \"75d0141a-4aa4-41fe-b47d-1e564cc52302\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" Oct 30 09:39:11.981982 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:11.981852 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d0141a-4aa4-41fe-b47d-1e564cc52302-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh\" (UID: \"75d0141a-4aa4-41fe-b47d-1e564cc52302\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" Oct 30 09:39:12.004405 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:12.004374 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" Oct 30 09:39:12.086661 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:12.083101 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbmb8\" (UniqueName: \"kubernetes.io/projected/75d0141a-4aa4-41fe-b47d-1e564cc52302-kube-api-access-dbmb8\") pod \"payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh\" (UID: \"75d0141a-4aa4-41fe-b47d-1e564cc52302\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" Oct 30 09:39:12.086661 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:12.083197 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d0141a-4aa4-41fe-b47d-1e564cc52302-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh\" (UID: \"75d0141a-4aa4-41fe-b47d-1e564cc52302\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" Oct 30 09:39:12.086661 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:12.084334 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d0141a-4aa4-41fe-b47d-1e564cc52302-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh\" (UID: \"75d0141a-4aa4-41fe-b47d-1e564cc52302\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" Oct 30 09:39:12.107785 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:12.107707 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbmb8\" (UniqueName: \"kubernetes.io/projected/75d0141a-4aa4-41fe-b47d-1e564cc52302-kube-api-access-dbmb8\") pod \"payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh\" (UID: \"75d0141a-4aa4-41fe-b47d-1e564cc52302\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" Oct 30 09:39:12.108529 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:12.108300 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" Oct 30 09:39:12.306225 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:12.306201 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks"] Oct 30 09:39:12.307104 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:39:12.307081 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a880792_dae1_47f4_96e9_7df47ce563cf.slice/crio-e710c2bee8fa438779d9e188ce779200ec5c46206ab29f8c17224b77ec47a79d WatchSource:0}: Error finding container e710c2bee8fa438779d9e188ce779200ec5c46206ab29f8c17224b77ec47a79d: Status 404 returned error can't find the container with id e710c2bee8fa438779d9e188ce779200ec5c46206ab29f8c17224b77ec47a79d Oct 30 09:39:12.420342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:12.420288 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh"] Oct 30 09:39:12.422556 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:39:12.422522 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d0141a_4aa4_41fe_b47d_1e564cc52302.slice/crio-96476d8bd403f328e0fc0409da650a5d41feaa280ec4b364e28130c1eec1732c WatchSource:0}: Error finding container 96476d8bd403f328e0fc0409da650a5d41feaa280ec4b364e28130c1eec1732c: Status 404 returned error can't find the container with id 96476d8bd403f328e0fc0409da650a5d41feaa280ec4b364e28130c1eec1732c Oct 30 09:39:12.657193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:12.657147 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" event={"ID":"7a880792-dae1-47f4-96e9-7df47ce563cf","Type":"ContainerStarted","Data":"bc4f67bcbfa99fa4056c91f24b995633800803385e7c0ffa3bfe5d100ced17d0"} Oct 30 09:39:12.657711 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:12.657214 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" event={"ID":"7a880792-dae1-47f4-96e9-7df47ce563cf","Type":"ContainerStarted","Data":"e710c2bee8fa438779d9e188ce779200ec5c46206ab29f8c17224b77ec47a79d"} Oct 30 09:39:12.657711 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:12.657342 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" Oct 30 09:39:12.658995 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:12.658972 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" event={"ID":"75d0141a-4aa4-41fe-b47d-1e564cc52302","Type":"ContainerStarted","Data":"ca0bc92ce1d02eabd7d874e83ab85ebea78e16e43bc38589b4b3dab4ae500f78"} Oct 30 09:39:12.659096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:12.659010 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" event={"ID":"75d0141a-4aa4-41fe-b47d-1e564cc52302","Type":"ContainerStarted","Data":"96476d8bd403f328e0fc0409da650a5d41feaa280ec4b364e28130c1eec1732c"} Oct 30 09:39:12.659096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:12.659030 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" Oct 30 09:39:12.701105 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:12.701022 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" podStartSLOduration=1.7010082949999998 podStartE2EDuration="1.701008295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:12.698274682 +0000 UTC m=+2670.560980502" watchObservedRunningTime="2026-10-30 09:39:12.701008295 +0000 UTC m=+2670.563714118" Oct 30 09:39:12.742163 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:12.742113 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" podStartSLOduration=1.7420957700000002 podStartE2EDuration="1.74209577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:12.738727949 +0000 UTC m=+2670.601433772" watchObservedRunningTime="2026-10-30 09:39:12.74209577 +0000 UTC m=+2670.604801591" Oct 30 09:39:13.429192 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:13.429139 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns"] Oct 30 09:39:15.814843 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:39:15.814795 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:39:21.206924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:21.206880 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z"] Oct 30 09:39:21.207417 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:21.207167 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" podUID="28c15e50-b4b3-4b40-92a2-a0f392f2e63e" containerName="istio-proxy" containerID="cri-o://b8a4749a111d2d17fd442bec7e4b200d1534abcc4849e4a4049aeaf98b486157" gracePeriod=30 Oct 30 09:39:22.045234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:22.045187 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq"] Oct 30 09:39:22.049991 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:22.049967 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" Oct 30 09:39:22.067950 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:22.067917 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-6d4b7a\"" Oct 30 09:39:22.068214 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:22.067968 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-6d4b7a-dockercfg-mpvqw\"" Oct 30 09:39:22.071113 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:22.071086 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq"] Oct 30 09:39:22.178612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:22.177563 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsk7h\" (UniqueName: \"kubernetes.io/projected/bc478189-5da9-4f6b-9bdc-c713894f061d-kube-api-access-dsk7h\") pod \"payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq\" (UID: \"bc478189-5da9-4f6b-9bdc-c713894f061d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" Oct 30 09:39:22.178612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:22.177930 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc478189-5da9-4f6b-9bdc-c713894f061d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq\" (UID: \"bc478189-5da9-4f6b-9bdc-c713894f061d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" Oct 30 09:39:22.280633 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:22.280574 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc478189-5da9-4f6b-9bdc-c713894f061d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq\" (UID: \"bc478189-5da9-4f6b-9bdc-c713894f061d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" Oct 30 09:39:22.281241 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:22.280669 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsk7h\" (UniqueName: \"kubernetes.io/projected/bc478189-5da9-4f6b-9bdc-c713894f061d-kube-api-access-dsk7h\") pod \"payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq\" (UID: \"bc478189-5da9-4f6b-9bdc-c713894f061d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" Oct 30 09:39:22.282157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:22.282132 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc478189-5da9-4f6b-9bdc-c713894f061d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq\" (UID: \"bc478189-5da9-4f6b-9bdc-c713894f061d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" Oct 30 09:39:22.304487 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:22.304403 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsk7h\" (UniqueName: \"kubernetes.io/projected/bc478189-5da9-4f6b-9bdc-c713894f061d-kube-api-access-dsk7h\") pod \"payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq\" (UID: \"bc478189-5da9-4f6b-9bdc-c713894f061d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" Oct 30 09:39:22.365807 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:22.365768 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" Oct 30 09:39:22.672412 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:22.672382 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq"] Oct 30 09:39:22.673789 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:39:22.673759 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc478189_5da9_4f6b_9bdc_c713894f061d.slice/crio-15c871eefef7945b9b8f6b03f8e8ea12eb61abc0d93567370c37b00e77b8fca9 WatchSource:0}: Error finding container 15c871eefef7945b9b8f6b03f8e8ea12eb61abc0d93567370c37b00e77b8fca9: Status 404 returned error can't find the container with id 15c871eefef7945b9b8f6b03f8e8ea12eb61abc0d93567370c37b00e77b8fca9 Oct 30 09:39:22.734031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:22.733997 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" event={"ID":"bc478189-5da9-4f6b-9bdc-c713894f061d","Type":"ContainerStarted","Data":"15c871eefef7945b9b8f6b03f8e8ea12eb61abc0d93567370c37b00e77b8fca9"} Oct 30 09:39:23.669080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:23.669054 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" Oct 30 09:39:23.669546 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:23.669127 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" Oct 30 09:39:23.746062 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:23.746022 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" event={"ID":"bc478189-5da9-4f6b-9bdc-c713894f061d","Type":"ContainerStarted","Data":"d583fcdf7165adf7cfb6152f6b39a004f2bd9652db43cc0ce0c2937b12585862"} Oct 30 09:39:23.746242 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:23.746138 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" Oct 30 09:39:23.795829 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:23.795774 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" podStartSLOduration=1.795758927 podStartE2EDuration="1.795758927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:23.793629858 +0000 UTC m=+2681.656335692" watchObservedRunningTime="2026-10-30 09:39:23.795758927 +0000 UTC m=+2681.658464746" Oct 30 09:39:24.576689 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:24.576655 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46"] Oct 30 09:39:24.576890 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:24.576865 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" podUID="558ecb6f-0981-4211-8768-1c5de58945b8" containerName="istio-proxy" containerID="cri-o://82f80820a2c48ad871f7b239205c03b46d130d780b54cbfa5e0eefa9a839ecee" gracePeriod=30 Oct 30 09:39:26.486622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.486542 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:39:26.646044 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.645993 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-data\") pod \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " Oct 30 09:39:26.646238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.646061 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-envoy\") pod \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " Oct 30 09:39:26.646238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.646182 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-workload-certs\") pod \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " Oct 30 09:39:26.646238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.646219 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istiod-ca-cert\") pod \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " Oct 30 09:39:26.646427 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.646266 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-podinfo\") pod \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " Oct 30 09:39:26.646427 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.646312 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-data" pod "28c15e50-b4b3-4b40-92a2-a0f392f2e63e" (UID: "28c15e50-b4b3-4b40-92a2-a0f392f2e63e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:26.646427 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.646383 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-kube-api-access-jqgdr\" (UniqueName: \"kubernetes.io/projected/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-kube-api-access-jqgdr\") pod \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " Oct 30 09:39:26.646427 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.646421 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-credential-socket\") pod \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " Oct 30 09:39:26.646693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.646482 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-workload-socket\") pod \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " Oct 30 09:39:26.646693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.646532 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-token\" (UniqueName: \"kubernetes.io/projected/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-token\") pod \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\" (UID: \"28c15e50-b4b3-4b40-92a2-a0f392f2e63e\") " Oct 30 09:39:26.647104 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.646531 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-workload-certs" pod "28c15e50-b4b3-4b40-92a2-a0f392f2e63e" (UID: "28c15e50-b4b3-4b40-92a2-a0f392f2e63e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:26.647104 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.646752 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-workload-socket" pod "28c15e50-b4b3-4b40-92a2-a0f392f2e63e" (UID: "28c15e50-b4b3-4b40-92a2-a0f392f2e63e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:26.647104 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.646920 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-credential-socket" pod "28c15e50-b4b3-4b40-92a2-a0f392f2e63e" (UID: "28c15e50-b4b3-4b40-92a2-a0f392f2e63e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:26.647104 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.646967 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istiod-ca-cert" pod "28c15e50-b4b3-4b40-92a2-a0f392f2e63e" (UID: "28c15e50-b4b3-4b40-92a2-a0f392f2e63e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:39:26.647542 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.647519 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.647653 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.647548 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.647653 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.647563 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.647653 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.647576 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.647653 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.647609 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.648793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.648767 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-envoy" pod "28c15e50-b4b3-4b40-92a2-a0f392f2e63e" (UID: "28c15e50-b4b3-4b40-92a2-a0f392f2e63e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:26.649040 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.649010 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-podinfo" pod "28c15e50-b4b3-4b40-92a2-a0f392f2e63e" (UID: "28c15e50-b4b3-4b40-92a2-a0f392f2e63e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:39:26.649120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.649092 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-kube-api-access-jqgdr" pod "28c15e50-b4b3-4b40-92a2-a0f392f2e63e" (UID: "28c15e50-b4b3-4b40-92a2-a0f392f2e63e"). InnerVolumeSpecName "kube-api-access-jqgdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:39:26.649325 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.649304 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-token" pod "28c15e50-b4b3-4b40-92a2-a0f392f2e63e" (UID: "28c15e50-b4b3-4b40-92a2-a0f392f2e63e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:39:26.748691 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.748607 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.748691 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.748635 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.748691 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.748647 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqgdr\" (UniqueName: \"kubernetes.io/projected/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-kube-api-access-jqgdr\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.748691 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.748656 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c15e50-b4b3-4b40-92a2-a0f392f2e63e-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.769768 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.769727 5938 generic.go:356] "Generic (PLEG): container finished" podID="28c15e50-b4b3-4b40-92a2-a0f392f2e63e" containerID="b8a4749a111d2d17fd442bec7e4b200d1534abcc4849e4a4049aeaf98b486157" exitCode=0 Oct 30 09:39:26.769958 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.769784 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" Oct 30 09:39:26.769958 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.769778 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" event={"ID":"28c15e50-b4b3-4b40-92a2-a0f392f2e63e","Type":"ContainerDied","Data":"b8a4749a111d2d17fd442bec7e4b200d1534abcc4849e4a4049aeaf98b486157"} Oct 30 09:39:26.769958 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.769921 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z" event={"ID":"28c15e50-b4b3-4b40-92a2-a0f392f2e63e","Type":"ContainerDied","Data":"0485bb271c58ec8ad09971cca5a7cee8fb3ba3fc286aa86527268fb1424d624e"} Oct 30 09:39:26.770131 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.769952 5938 scope.go:122] "RemoveContainer" containerID="b8a4749a111d2d17fd442bec7e4b200d1534abcc4849e4a4049aeaf98b486157" Oct 30 09:39:26.781483 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.781466 5938 scope.go:122] "RemoveContainer" containerID="b8a4749a111d2d17fd442bec7e4b200d1534abcc4849e4a4049aeaf98b486157" Oct 30 09:39:26.781890 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:39:26.781869 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a4749a111d2d17fd442bec7e4b200d1534abcc4849e4a4049aeaf98b486157\": container with ID starting with b8a4749a111d2d17fd442bec7e4b200d1534abcc4849e4a4049aeaf98b486157 not found: ID does not exist" containerID="b8a4749a111d2d17fd442bec7e4b200d1534abcc4849e4a4049aeaf98b486157" Oct 30 09:39:26.781979 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.781895 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a4749a111d2d17fd442bec7e4b200d1534abcc4849e4a4049aeaf98b486157"} err="failed to get container status \"b8a4749a111d2d17fd442bec7e4b200d1534abcc4849e4a4049aeaf98b486157\": rpc error: code = NotFound desc = could not find container \"b8a4749a111d2d17fd442bec7e4b200d1534abcc4849e4a4049aeaf98b486157\": container with ID starting with b8a4749a111d2d17fd442bec7e4b200d1534abcc4849e4a4049aeaf98b486157 not found: ID does not exist" Oct 30 09:39:26.826654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.826621 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z"] Oct 30 09:39:26.832782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:26.832545 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-pmm5z"] Oct 30 09:39:27.159047 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:27.159011 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc"] Oct 30 09:39:27.159273 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:27.159218 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" podUID="3aa20f8e-6be0-484a-a363-28bbe0aa6d88" containerName="istio-proxy" containerID="cri-o://d34cff2ccf703aab7192adf5699bc305b19824f44074030efbb0b5f08779c024" gracePeriod=30 Oct 30 09:39:28.261369 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.261332 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c15e50-b4b3-4b40-92a2-a0f392f2e63e" path="/var/lib/kubelet/pods/28c15e50-b4b3-4b40-92a2-a0f392f2e63e/volumes" Oct 30 09:39:28.455527 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.455479 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" podUID="4cd25eb1-3de8-48ff-92ae-06d486f584b6" containerName="main" containerID="cri-o://ccd895375d0a741aea70b2a5fe18615ac0e9ff6a558bcae7077e6bf463c84b9a" gracePeriod=45 Oct 30 09:39:28.563665 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.556987 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" podUID="4cd25eb1-3de8-48ff-92ae-06d486f584b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.137:8000/ready\": dial tcp 10.134.0.137:8000: connect: connection refused" Oct 30 09:39:28.774781 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.774757 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:39:28.779550 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.779522 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd25eb1-3de8-48ff-92ae-06d486f584b6-kube-api-access-zckb7\" (UniqueName: \"kubernetes.io/projected/4cd25eb1-3de8-48ff-92ae-06d486f584b6-kube-api-access-zckb7\") pod \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " Oct 30 09:39:28.779711 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.779640 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cd25eb1-3de8-48ff-92ae-06d486f584b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/4cd25eb1-3de8-48ff-92ae-06d486f584b6-tls-certs\") pod \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " Oct 30 09:39:28.779806 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.779715 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-tmp-dir\") pod \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " Oct 30 09:39:28.779806 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.779791 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-model-cache\") pod \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " Oct 30 09:39:28.779923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.779885 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-dshm\") pod \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " Oct 30 09:39:28.779923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.779910 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-home\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-home\") pod \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\" (UID: \"4cd25eb1-3de8-48ff-92ae-06d486f584b6\") " Oct 30 09:39:28.780399 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.780077 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-tmp-dir" pod "4cd25eb1-3de8-48ff-92ae-06d486f584b6" (UID: "4cd25eb1-3de8-48ff-92ae-06d486f584b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:28.780399 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.780160 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-model-cache" pod "4cd25eb1-3de8-48ff-92ae-06d486f584b6" (UID: "4cd25eb1-3de8-48ff-92ae-06d486f584b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:28.780691 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.780639 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-home" pod "4cd25eb1-3de8-48ff-92ae-06d486f584b6" (UID: "4cd25eb1-3de8-48ff-92ae-06d486f584b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:28.780760 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.780731 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:28.780760 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.780752 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:28.783881 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.783459 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cd25eb1-3de8-48ff-92ae-06d486f584b6-tls-certs" pod "4cd25eb1-3de8-48ff-92ae-06d486f584b6" (UID: "4cd25eb1-3de8-48ff-92ae-06d486f584b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:39:28.783881 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.783531 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-dshm" pod "4cd25eb1-3de8-48ff-92ae-06d486f584b6" (UID: "4cd25eb1-3de8-48ff-92ae-06d486f584b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:28.783881 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.783711 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd25eb1-3de8-48ff-92ae-06d486f584b6-kube-api-access-zckb7" pod "4cd25eb1-3de8-48ff-92ae-06d486f584b6" (UID: "4cd25eb1-3de8-48ff-92ae-06d486f584b6"). InnerVolumeSpecName "kube-api-access-zckb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:39:28.792499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.792462 5938 generic.go:356] "Generic (PLEG): container finished" podID="4cd25eb1-3de8-48ff-92ae-06d486f584b6" containerID="ccd895375d0a741aea70b2a5fe18615ac0e9ff6a558bcae7077e6bf463c84b9a" exitCode=0 Oct 30 09:39:28.792679 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.792529 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" event={"ID":"4cd25eb1-3de8-48ff-92ae-06d486f584b6","Type":"ContainerDied","Data":"ccd895375d0a741aea70b2a5fe18615ac0e9ff6a558bcae7077e6bf463c84b9a"} Oct 30 09:39:28.792679 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.792579 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" event={"ID":"4cd25eb1-3de8-48ff-92ae-06d486f584b6","Type":"ContainerDied","Data":"516a32d1fa92c946066108d36d765d47950c06fc106d2fb1c4caa7e60dd50435"} Oct 30 09:39:28.792679 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.792604 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns" Oct 30 09:39:28.792679 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.792614 5938 scope.go:122] "RemoveContainer" containerID="ccd895375d0a741aea70b2a5fe18615ac0e9ff6a558bcae7077e6bf463c84b9a" Oct 30 09:39:28.834251 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.834207 5938 scope.go:122] "RemoveContainer" containerID="ccd895375d0a741aea70b2a5fe18615ac0e9ff6a558bcae7077e6bf463c84b9a" Oct 30 09:39:28.834924 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:39:28.834895 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd895375d0a741aea70b2a5fe18615ac0e9ff6a558bcae7077e6bf463c84b9a\": container with ID starting with ccd895375d0a741aea70b2a5fe18615ac0e9ff6a558bcae7077e6bf463c84b9a not found: ID does not exist" containerID="ccd895375d0a741aea70b2a5fe18615ac0e9ff6a558bcae7077e6bf463c84b9a" Oct 30 09:39:28.835072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.834934 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd895375d0a741aea70b2a5fe18615ac0e9ff6a558bcae7077e6bf463c84b9a"} err="failed to get container status \"ccd895375d0a741aea70b2a5fe18615ac0e9ff6a558bcae7077e6bf463c84b9a\": rpc error: code = NotFound desc = could not find container \"ccd895375d0a741aea70b2a5fe18615ac0e9ff6a558bcae7077e6bf463c84b9a\": container with ID starting with ccd895375d0a741aea70b2a5fe18615ac0e9ff6a558bcae7077e6bf463c84b9a not found: ID does not exist" Oct 30 09:39:28.880534 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.880492 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns"] Oct 30 09:39:28.882411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.882380 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:28.882411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.882411 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cd25eb1-3de8-48ff-92ae-06d486f584b6-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:28.882578 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.882421 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zckb7\" (UniqueName: \"kubernetes.io/projected/4cd25eb1-3de8-48ff-92ae-06d486f584b6-kube-api-access-zckb7\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:28.882578 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.882432 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cd25eb1-3de8-48ff-92ae-06d486f584b6-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:28.887076 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:28.887044 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-304459/auth-test-model-304459-kserve-7cb68f67c-wz9ns"] Oct 30 09:39:29.249452 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.249419 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4ln46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:39:29.249678 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.249469 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" podUID="558ecb6f-0981-4211-8768-1c5de58945b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:39:29.803306 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.803273 5938 generic.go:356] "Generic (PLEG): container finished" podID="558ecb6f-0981-4211-8768-1c5de58945b8" containerID="82f80820a2c48ad871f7b239205c03b46d130d780b54cbfa5e0eefa9a839ecee" exitCode=0 Oct 30 09:39:29.803857 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.803400 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" event={"ID":"558ecb6f-0981-4211-8768-1c5de58945b8","Type":"ContainerDied","Data":"82f80820a2c48ad871f7b239205c03b46d130d780b54cbfa5e0eefa9a839ecee"} Oct 30 09:39:29.851988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.851913 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:39:29.896409 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.896379 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558ecb6f-0981-4211-8768-1c5de58945b8-istio-token\" (UniqueName: \"kubernetes.io/projected/558ecb6f-0981-4211-8768-1c5de58945b8-istio-token\") pod \"558ecb6f-0981-4211-8768-1c5de58945b8\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " Oct 30 09:39:29.896639 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.896447 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-workload-certs\") pod \"558ecb6f-0981-4211-8768-1c5de58945b8\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " Oct 30 09:39:29.896639 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.896510 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558ecb6f-0981-4211-8768-1c5de58945b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558ecb6f-0981-4211-8768-1c5de58945b8-istiod-ca-cert\") pod \"558ecb6f-0981-4211-8768-1c5de58945b8\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " Oct 30 09:39:29.896639 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.896551 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558ecb6f-0981-4211-8768-1c5de58945b8-kube-api-access-5jkkh\" (UniqueName: \"kubernetes.io/projected/558ecb6f-0981-4211-8768-1c5de58945b8-kube-api-access-5jkkh\") pod \"558ecb6f-0981-4211-8768-1c5de58945b8\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " Oct 30 09:39:29.896814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.896663 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-credential-socket\") pod \"558ecb6f-0981-4211-8768-1c5de58945b8\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " Oct 30 09:39:29.896814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.896736 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-workload-certs" pod "558ecb6f-0981-4211-8768-1c5de58945b8" (UID: "558ecb6f-0981-4211-8768-1c5de58945b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:29.896924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.896816 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-istio-data\") pod \"558ecb6f-0981-4211-8768-1c5de58945b8\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " Oct 30 09:39:29.896924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.896860 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-workload-socket\") pod \"558ecb6f-0981-4211-8768-1c5de58945b8\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " Oct 30 09:39:29.896924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.896897 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-istio-envoy\") pod \"558ecb6f-0981-4211-8768-1c5de58945b8\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " Oct 30 09:39:29.897080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.896925 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-credential-socket" pod "558ecb6f-0981-4211-8768-1c5de58945b8" (UID: "558ecb6f-0981-4211-8768-1c5de58945b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:29.897080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.896985 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/558ecb6f-0981-4211-8768-1c5de58945b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558ecb6f-0981-4211-8768-1c5de58945b8-istio-podinfo\") pod \"558ecb6f-0981-4211-8768-1c5de58945b8\" (UID: \"558ecb6f-0981-4211-8768-1c5de58945b8\") " Oct 30 09:39:29.897080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.897008 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558ecb6f-0981-4211-8768-1c5de58945b8-istiod-ca-cert" pod "558ecb6f-0981-4211-8768-1c5de58945b8" (UID: "558ecb6f-0981-4211-8768-1c5de58945b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:39:29.897242 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.897087 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-istio-data" pod "558ecb6f-0981-4211-8768-1c5de58945b8" (UID: "558ecb6f-0981-4211-8768-1c5de58945b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:29.897242 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.897206 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-workload-socket" pod "558ecb6f-0981-4211-8768-1c5de58945b8" (UID: "558ecb6f-0981-4211-8768-1c5de58945b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:29.897560 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.897540 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:29.897676 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.897565 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558ecb6f-0981-4211-8768-1c5de58945b8-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:29.897676 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.897578 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:29.897676 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.897625 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:29.897676 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.897639 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:29.898996 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.898976 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558ecb6f-0981-4211-8768-1c5de58945b8-kube-api-access-5jkkh" pod "558ecb6f-0981-4211-8768-1c5de58945b8" (UID: "558ecb6f-0981-4211-8768-1c5de58945b8"). InnerVolumeSpecName "kube-api-access-5jkkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:39:29.899238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.899224 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/558ecb6f-0981-4211-8768-1c5de58945b8-istio-podinfo" pod "558ecb6f-0981-4211-8768-1c5de58945b8" (UID: "558ecb6f-0981-4211-8768-1c5de58945b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:39:29.899451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.899435 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558ecb6f-0981-4211-8768-1c5de58945b8-istio-token" pod "558ecb6f-0981-4211-8768-1c5de58945b8" (UID: "558ecb6f-0981-4211-8768-1c5de58945b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:39:29.899490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.899467 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-istio-envoy" pod "558ecb6f-0981-4211-8768-1c5de58945b8" (UID: "558ecb6f-0981-4211-8768-1c5de58945b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:29.999327 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.999291 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jkkh\" (UniqueName: \"kubernetes.io/projected/558ecb6f-0981-4211-8768-1c5de58945b8-kube-api-access-5jkkh\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:29.999327 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.999322 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558ecb6f-0981-4211-8768-1c5de58945b8-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:29.999327 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.999332 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558ecb6f-0981-4211-8768-1c5de58945b8-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:29.999570 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:29.999340 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558ecb6f-0981-4211-8768-1c5de58945b8-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:30.258618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:30.258551 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd25eb1-3de8-48ff-92ae-06d486f584b6" path="/var/lib/kubelet/pods/4cd25eb1-3de8-48ff-92ae-06d486f584b6/volumes" Oct 30 09:39:30.814210 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:30.814173 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" event={"ID":"558ecb6f-0981-4211-8768-1c5de58945b8","Type":"ContainerDied","Data":"9e5e20cb3139805c17101f685a4986e83f7661c345912f592cd86f0a04ada2ee"} Oct 30 09:39:30.814768 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:30.814228 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46" Oct 30 09:39:30.814768 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:30.814235 5938 scope.go:122] "RemoveContainer" containerID="82f80820a2c48ad871f7b239205c03b46d130d780b54cbfa5e0eefa9a839ecee" Oct 30 09:39:30.887973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:30.887932 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46"] Oct 30 09:39:30.901063 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:30.901029 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4ln46"] Oct 30 09:39:32.258365 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.258328 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="558ecb6f-0981-4211-8768-1c5de58945b8" path="/var/lib/kubelet/pods/558ecb6f-0981-4211-8768-1c5de58945b8/volumes" Oct 30 09:39:32.430455 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.430384 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:39:32.531236 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.531200 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-podinfo\") pod \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " Oct 30 09:39:32.531443 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.531265 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-workload-certs\") pod \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " Oct 30 09:39:32.531443 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.531307 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istiod-ca-cert\") pod \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " Oct 30 09:39:32.531443 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.531348 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-credential-socket\") pod \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " Oct 30 09:39:32.531443 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.531383 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-token\" (UniqueName: \"kubernetes.io/projected/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-token\") pod \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " Oct 30 09:39:32.531735 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.531709 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-kube-api-access-xqcm8\" (UniqueName: \"kubernetes.io/projected/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-kube-api-access-xqcm8\") pod \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " Oct 30 09:39:32.531812 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.531708 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-workload-certs" pod "3aa20f8e-6be0-484a-a363-28bbe0aa6d88" (UID: "3aa20f8e-6be0-484a-a363-28bbe0aa6d88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:32.531863 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.531785 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istiod-ca-cert" pod "3aa20f8e-6be0-484a-a363-28bbe0aa6d88" (UID: "3aa20f8e-6be0-484a-a363-28bbe0aa6d88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:39:32.531863 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.531795 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-data\") pod \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " Oct 30 09:39:32.531967 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.531927 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-envoy\") pod \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " Oct 30 09:39:32.532090 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.531979 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-credential-socket" pod "3aa20f8e-6be0-484a-a363-28bbe0aa6d88" (UID: "3aa20f8e-6be0-484a-a363-28bbe0aa6d88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:32.532090 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.532042 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-data" pod "3aa20f8e-6be0-484a-a363-28bbe0aa6d88" (UID: "3aa20f8e-6be0-484a-a363-28bbe0aa6d88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:32.532436 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.532374 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-workload-socket\") pod \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\" (UID: \"3aa20f8e-6be0-484a-a363-28bbe0aa6d88\") " Oct 30 09:39:32.533012 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.532987 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.533104 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.533014 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.533104 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.533028 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.533104 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.533042 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.533314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.533291 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-workload-socket" pod "3aa20f8e-6be0-484a-a363-28bbe0aa6d88" (UID: "3aa20f8e-6be0-484a-a363-28bbe0aa6d88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:32.533965 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.533940 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-kube-api-access-xqcm8" pod "3aa20f8e-6be0-484a-a363-28bbe0aa6d88" (UID: "3aa20f8e-6be0-484a-a363-28bbe0aa6d88"). InnerVolumeSpecName "kube-api-access-xqcm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:39:32.534186 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.534073 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-podinfo" pod "3aa20f8e-6be0-484a-a363-28bbe0aa6d88" (UID: "3aa20f8e-6be0-484a-a363-28bbe0aa6d88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:39:32.534186 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.534159 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-token" pod "3aa20f8e-6be0-484a-a363-28bbe0aa6d88" (UID: "3aa20f8e-6be0-484a-a363-28bbe0aa6d88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:39:32.534696 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.534675 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-envoy" pod "3aa20f8e-6be0-484a-a363-28bbe0aa6d88" (UID: "3aa20f8e-6be0-484a-a363-28bbe0aa6d88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:32.634540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.634507 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.634540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.634539 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.634540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.634550 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.634802 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.634559 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.634802 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.634568 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqcm8\" (UniqueName: \"kubernetes.io/projected/3aa20f8e-6be0-484a-a363-28bbe0aa6d88-kube-api-access-xqcm8\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.831926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.831831 5938 generic.go:356] "Generic (PLEG): container finished" podID="3aa20f8e-6be0-484a-a363-28bbe0aa6d88" containerID="d34cff2ccf703aab7192adf5699bc305b19824f44074030efbb0b5f08779c024" exitCode=0 Oct 30 09:39:32.831926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.831880 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" Oct 30 09:39:32.831926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.831907 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" event={"ID":"3aa20f8e-6be0-484a-a363-28bbe0aa6d88","Type":"ContainerDied","Data":"d34cff2ccf703aab7192adf5699bc305b19824f44074030efbb0b5f08779c024"} Oct 30 09:39:32.832199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.831966 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc" event={"ID":"3aa20f8e-6be0-484a-a363-28bbe0aa6d88","Type":"ContainerDied","Data":"bcda93dd789c893838130f3b9a2b170b516b4e6d5d45a1ee1ac91069ee4ce2bd"} Oct 30 09:39:32.832199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.831990 5938 scope.go:122] "RemoveContainer" containerID="d34cff2ccf703aab7192adf5699bc305b19824f44074030efbb0b5f08779c024" Oct 30 09:39:32.844173 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.844154 5938 scope.go:122] "RemoveContainer" containerID="d34cff2ccf703aab7192adf5699bc305b19824f44074030efbb0b5f08779c024" Oct 30 09:39:32.844467 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:39:32.844449 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34cff2ccf703aab7192adf5699bc305b19824f44074030efbb0b5f08779c024\": container with ID starting with d34cff2ccf703aab7192adf5699bc305b19824f44074030efbb0b5f08779c024 not found: ID does not exist" containerID="d34cff2ccf703aab7192adf5699bc305b19824f44074030efbb0b5f08779c024" Oct 30 09:39:32.844532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.844479 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34cff2ccf703aab7192adf5699bc305b19824f44074030efbb0b5f08779c024"} err="failed to get container status \"d34cff2ccf703aab7192adf5699bc305b19824f44074030efbb0b5f08779c024\": rpc error: code = NotFound desc = could not find container \"d34cff2ccf703aab7192adf5699bc305b19824f44074030efbb0b5f08779c024\": container with ID starting with d34cff2ccf703aab7192adf5699bc305b19824f44074030efbb0b5f08779c024 not found: ID does not exist" Oct 30 09:39:32.896508 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.896476 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc"] Oct 30 09:39:32.909488 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:32.909455 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-gv9nc"] Oct 30 09:39:34.261001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.260951 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa20f8e-6be0-484a-a363-28bbe0aa6d88" path="/var/lib/kubelet/pods/3aa20f8e-6be0-484a-a363-28bbe0aa6d88/volumes" Oct 30 09:39:34.528300 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.528201 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj"] Oct 30 09:39:34.528924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.528906 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3aa20f8e-6be0-484a-a363-28bbe0aa6d88" containerName="istio-proxy" Oct 30 09:39:34.528924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.528925 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa20f8e-6be0-484a-a363-28bbe0aa6d88" containerName="istio-proxy" Oct 30 09:39:34.529110 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.528936 5938 cpu_manager.go:414] "containerMap: removing container" podUID="558ecb6f-0981-4211-8768-1c5de58945b8" containerName="istio-proxy" Oct 30 09:39:34.529110 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.528941 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558ecb6f-0981-4211-8768-1c5de58945b8" containerName="istio-proxy" Oct 30 09:39:34.529110 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.528987 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4cd25eb1-3de8-48ff-92ae-06d486f584b6" containerName="main" Oct 30 09:39:34.529110 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.528994 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd25eb1-3de8-48ff-92ae-06d486f584b6" containerName="main" Oct 30 09:39:34.529110 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.529005 5938 cpu_manager.go:414] "containerMap: removing container" podUID="28c15e50-b4b3-4b40-92a2-a0f392f2e63e" containerName="istio-proxy" Oct 30 09:39:34.529110 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.529010 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c15e50-b4b3-4b40-92a2-a0f392f2e63e" containerName="istio-proxy" Oct 30 09:39:34.529342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.529119 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa20f8e-6be0-484a-a363-28bbe0aa6d88" containerName="istio-proxy" Oct 30 09:39:34.529342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.529131 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="558ecb6f-0981-4211-8768-1c5de58945b8" containerName="istio-proxy" Oct 30 09:39:34.529342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.529150 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd25eb1-3de8-48ff-92ae-06d486f584b6" containerName="main" Oct 30 09:39:34.529342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.529161 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c15e50-b4b3-4b40-92a2-a0f392f2e63e" containerName="istio-proxy" Oct 30 09:39:34.534363 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.534343 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" Oct 30 09:39:34.551325 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.551295 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj"] Oct 30 09:39:34.561328 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.561289 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-035d7d\"" Oct 30 09:39:34.573107 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.573066 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcxfw\" (UniqueName: \"kubernetes.io/projected/1aa4b129-c86e-4749-9a1a-bcda6f840722-kube-api-access-pcxfw\") pod \"payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj\" (UID: \"1aa4b129-c86e-4749-9a1a-bcda6f840722\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" Oct 30 09:39:34.573285 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.573187 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa4b129-c86e-4749-9a1a-bcda6f840722-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj\" (UID: \"1aa4b129-c86e-4749-9a1a-bcda6f840722\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" Oct 30 09:39:34.595622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.594685 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-035d7d-dockercfg-p2dk9\"" Oct 30 09:39:34.605605 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.605548 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n"] Oct 30 09:39:34.610986 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.610953 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" Oct 30 09:39:34.662613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.635998 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n"] Oct 30 09:39:34.684201 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.680561 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2f96b2-b18a-4ede-8444-312790bde84c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n\" (UID: \"2d2f96b2-b18a-4ede-8444-312790bde84c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" Oct 30 09:39:34.684201 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.680723 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcxfw\" (UniqueName: \"kubernetes.io/projected/1aa4b129-c86e-4749-9a1a-bcda6f840722-kube-api-access-pcxfw\") pod \"payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj\" (UID: \"1aa4b129-c86e-4749-9a1a-bcda6f840722\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" Oct 30 09:39:34.684201 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.680798 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa4b129-c86e-4749-9a1a-bcda6f840722-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj\" (UID: \"1aa4b129-c86e-4749-9a1a-bcda6f840722\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" Oct 30 09:39:34.684201 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.680920 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-772mq\" (UniqueName: \"kubernetes.io/projected/2d2f96b2-b18a-4ede-8444-312790bde84c-kube-api-access-772mq\") pod \"payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n\" (UID: \"2d2f96b2-b18a-4ede-8444-312790bde84c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" Oct 30 09:39:34.684201 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.682279 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa4b129-c86e-4749-9a1a-bcda6f840722-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj\" (UID: \"1aa4b129-c86e-4749-9a1a-bcda6f840722\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" Oct 30 09:39:34.705960 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.705874 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcxfw\" (UniqueName: \"kubernetes.io/projected/1aa4b129-c86e-4749-9a1a-bcda6f840722-kube-api-access-pcxfw\") pod \"payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj\" (UID: \"1aa4b129-c86e-4749-9a1a-bcda6f840722\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" Oct 30 09:39:34.756737 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.756712 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" Oct 30 09:39:34.782875 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.782785 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772mq\" (UniqueName: \"kubernetes.io/projected/2d2f96b2-b18a-4ede-8444-312790bde84c-kube-api-access-772mq\") pod \"payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n\" (UID: \"2d2f96b2-b18a-4ede-8444-312790bde84c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" Oct 30 09:39:34.783047 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.782908 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2f96b2-b18a-4ede-8444-312790bde84c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n\" (UID: \"2d2f96b2-b18a-4ede-8444-312790bde84c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" Oct 30 09:39:34.783722 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.783701 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2f96b2-b18a-4ede-8444-312790bde84c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n\" (UID: \"2d2f96b2-b18a-4ede-8444-312790bde84c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" Oct 30 09:39:34.804302 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.804277 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-772mq\" (UniqueName: \"kubernetes.io/projected/2d2f96b2-b18a-4ede-8444-312790bde84c-kube-api-access-772mq\") pod \"payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n\" (UID: \"2d2f96b2-b18a-4ede-8444-312790bde84c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" Oct 30 09:39:34.849167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.849132 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" Oct 30 09:39:34.986407 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:34.986373 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" Oct 30 09:39:35.162782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:35.162754 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj"] Oct 30 09:39:35.164080 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:39:35.164052 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa4b129_c86e_4749_9a1a_bcda6f840722.slice/crio-675745d3fa65a640b392e6956ce4f5ee16f35145c5011e704b3ccd83059d1e06 WatchSource:0}: Error finding container 675745d3fa65a640b392e6956ce4f5ee16f35145c5011e704b3ccd83059d1e06: Status 404 returned error can't find the container with id 675745d3fa65a640b392e6956ce4f5ee16f35145c5011e704b3ccd83059d1e06 Oct 30 09:39:35.293065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:35.292995 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n"] Oct 30 09:39:35.294346 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:39:35.294320 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2f96b2_b18a_4ede_8444_312790bde84c.slice/crio-bd082e42bdf371698cfeb32ad130464f08dff5bb7143b593c6d4c8d497f48485 WatchSource:0}: Error finding container bd082e42bdf371698cfeb32ad130464f08dff5bb7143b593c6d4c8d497f48485: Status 404 returned error can't find the container with id bd082e42bdf371698cfeb32ad130464f08dff5bb7143b593c6d4c8d497f48485 Oct 30 09:39:35.864981 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:35.864946 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" event={"ID":"2d2f96b2-b18a-4ede-8444-312790bde84c","Type":"ContainerStarted","Data":"f1807b01ddebf15c90f7a113fe11a9ba0d80e111baf12be12977a77b311d36fd"} Oct 30 09:39:35.865194 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:35.865007 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" event={"ID":"2d2f96b2-b18a-4ede-8444-312790bde84c","Type":"ContainerStarted","Data":"bd082e42bdf371698cfeb32ad130464f08dff5bb7143b593c6d4c8d497f48485"} Oct 30 09:39:35.865194 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:35.865037 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" Oct 30 09:39:35.866985 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:35.866957 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" event={"ID":"1aa4b129-c86e-4749-9a1a-bcda6f840722","Type":"ContainerStarted","Data":"d89781bfb28454598d383fde8cce45a2a20758c7ec3f00e5d87b06d5dc2f42eb"} Oct 30 09:39:35.867130 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:35.867011 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" event={"ID":"1aa4b129-c86e-4749-9a1a-bcda6f840722","Type":"ContainerStarted","Data":"675745d3fa65a640b392e6956ce4f5ee16f35145c5011e704b3ccd83059d1e06"} Oct 30 09:39:35.867130 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:35.867038 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" Oct 30 09:39:35.909164 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:35.909109 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" podStartSLOduration=1.90909383 podStartE2EDuration="1.90909383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:35.906958578 +0000 UTC m=+2693.769664401" watchObservedRunningTime="2026-10-30 09:39:35.90909383 +0000 UTC m=+2693.771799702" Oct 30 09:39:35.954894 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:35.954827 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" podStartSLOduration=1.954801973 podStartE2EDuration="1.954801973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:35.949344126 +0000 UTC m=+2693.812049970" watchObservedRunningTime="2026-10-30 09:39:35.954801973 +0000 UTC m=+2693.817507797" Oct 30 09:39:43.842662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:43.840225 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565"] Oct 30 09:39:43.847376 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:43.847343 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:43.866617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:43.866562 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-3d9b36-openshift-default-dockercfg-5ftvx\"" Oct 30 09:39:43.885380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:43.885341 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565"] Oct 30 09:39:43.918436 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:43.918395 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22nnt\" (UniqueName: \"kubernetes.io/projected/b9985d34-9195-47d7-8e4c-fb5a87ca969a-kube-api-access-22nnt\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:43.918676 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:43.918632 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-data\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:43.918766 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:43.918677 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-envoy\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:43.918838 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:43.918815 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-credential-socket\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:43.918905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:43.918877 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istiod-ca-cert\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:43.919003 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:43.918975 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-workload-socket\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:43.919107 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:43.919005 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-workload-certs\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:43.919107 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:43.919039 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-token\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:43.919194 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:43.919107 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-podinfo\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.020523 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.020477 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istiod-ca-cert\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.020768 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.020550 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-workload-socket\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.020768 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.020601 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-workload-certs\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.020768 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.020635 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-token\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.020768 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.020691 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-podinfo\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.020984 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.020830 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22nnt\" (UniqueName: \"kubernetes.io/projected/b9985d34-9195-47d7-8e4c-fb5a87ca969a-kube-api-access-22nnt\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.020984 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.020969 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-data\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.021068 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.021002 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-envoy\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.021068 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.021050 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-credential-socket\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.022741 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.021528 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-credential-socket\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.022741 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.022210 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istiod-ca-cert\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.022741 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.022455 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-workload-socket\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.022741 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.022713 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-workload-certs\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.028633 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.028381 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-data\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.032501 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.032415 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-envoy\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.032917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.032891 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-podinfo\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.049622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.049014 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-token\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.055620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.055261 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22nnt\" (UniqueName: \"kubernetes.io/projected/b9985d34-9195-47d7-8e4c-fb5a87ca969a-kube-api-access-22nnt\") pod \"e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.132986 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.132897 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s"] Oct 30 09:39:44.137921 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.137897 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:39:44.157905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.157870 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s"] Oct 30 09:39:44.164234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.164205 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:44.175859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.175819 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a525bd-serving-cert\"" Oct 30 09:39:44.224443 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.224411 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rngj2\" (UniqueName: \"kubernetes.io/projected/665293f1-a5e7-46a9-a797-9fee3027b9ba-kube-api-access-rngj2\") pod \"maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:39:44.224940 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.224465 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-metrics-certs\") pod \"maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:39:44.224940 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.224573 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:39:44.225208 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.224868 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/665293f1-a5e7-46a9-a797-9fee3027b9ba-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:39:44.326472 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.326424 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rngj2\" (UniqueName: \"kubernetes.io/projected/665293f1-a5e7-46a9-a797-9fee3027b9ba-kube-api-access-rngj2\") pod \"maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:39:44.326727 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.326495 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-metrics-certs\") pod \"maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:39:44.327165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.327131 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:39:44.327329 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:39:44.327299 5938 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-serving-cert: secret "maas-api-e2e-extmod-a-a525bd-serving-cert" not found Oct 30 09:39:44.327412 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.327352 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/665293f1-a5e7-46a9-a797-9fee3027b9ba-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:39:44.327412 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:39:44.327384 5938 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-maas-api-tls podName:665293f1-a5e7-46a9-a797-9fee3027b9ba nodeName:}" failed. No retries permitted until 2026-10-30 09:39:44.827361797 +0000 UTC m=+2702.690067615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-maas-api-tls") pod "maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" (UID: "665293f1-a5e7-46a9-a797-9fee3027b9ba") : secret "maas-api-e2e-extmod-a-a525bd-serving-cert" not found Oct 30 09:39:44.328138 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.328103 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/665293f1-a5e7-46a9-a797-9fee3027b9ba-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:39:44.329917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.329899 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-metrics-certs\") pod \"maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:39:44.356314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.356204 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rngj2\" (UniqueName: \"kubernetes.io/projected/665293f1-a5e7-46a9-a797-9fee3027b9ba-kube-api-access-rngj2\") pod \"maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:39:44.509726 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.509697 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565"] Oct 30 09:39:44.511103 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:39:44.511077 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9985d34_9195_47d7_8e4c_fb5a87ca969a.slice/crio-18277057b4f7404055cd4bdaa58931e4a3753ee83ca290317843892b05a95c88 WatchSource:0}: Error finding container 18277057b4f7404055cd4bdaa58931e4a3753ee83ca290317843892b05a95c88: Status 404 returned error can't find the container with id 18277057b4f7404055cd4bdaa58931e4a3753ee83ca290317843892b05a95c88 Oct 30 09:39:44.525016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.515820 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:39:44.525016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.516078 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:39:44.525016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.516297 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:39:44.835199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.835095 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:39:44.838081 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.838035 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:39:44.938044 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.938002 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" event={"ID":"b9985d34-9195-47d7-8e4c-fb5a87ca969a","Type":"ContainerStarted","Data":"062de02c198f95e5e1ae6f5449e1d2dc709521f9184312d820d57dc67dc7fbf9"} Oct 30 09:39:44.938553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.938114 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" event={"ID":"b9985d34-9195-47d7-8e4c-fb5a87ca969a","Type":"ContainerStarted","Data":"18277057b4f7404055cd4bdaa58931e4a3753ee83ca290317843892b05a95c88"} Oct 30 09:39:44.984775 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:44.984708 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" podStartSLOduration=1.9846893859999999 podStartE2EDuration="1.984689386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:44.982665383 +0000 UTC m=+2702.845371205" watchObservedRunningTime="2026-10-30 09:39:44.984689386 +0000 UTC m=+2702.847395217" Oct 30 09:39:45.053898 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:45.053860 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:39:45.165026 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:45.164948 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:45.183629 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:45.181985 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:45.413878 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:39:45.413847 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665293f1_a5e7_46a9_a797_9fee3027b9ba.slice/crio-8c807fda1387954c809776dd44f6da24cb8a78550e2237222e7678f3cc08a028 WatchSource:0}: Error finding container 8c807fda1387954c809776dd44f6da24cb8a78550e2237222e7678f3cc08a028: Status 404 returned error can't find the container with id 8c807fda1387954c809776dd44f6da24cb8a78550e2237222e7678f3cc08a028 Oct 30 09:39:45.414547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:45.414520 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s"] Oct 30 09:39:45.810676 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:39:45.810547 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:39:45.947541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:45.947509 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" event={"ID":"665293f1-a5e7-46a9-a797-9fee3027b9ba","Type":"ContainerStarted","Data":"8c807fda1387954c809776dd44f6da24cb8a78550e2237222e7678f3cc08a028"} Oct 30 09:39:45.947956 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:45.947769 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:45.948848 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:45.948827 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:39:46.877043 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:46.877016 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" Oct 30 09:39:46.877422 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:46.877402 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" Oct 30 09:39:46.958238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:46.958107 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" event={"ID":"665293f1-a5e7-46a9-a797-9fee3027b9ba","Type":"ContainerStarted","Data":"5bf44ce12094f98464dd3f388135ca3efab01e51a0fd1b54569ad907a1eeca52"} Oct 30 09:39:46.958817 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:46.958794 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:39:47.037733 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:47.037679 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" podStartSLOduration=2.560687958 podStartE2EDuration="3.037662342s" totalImagesPullingTime="476.974384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:47.036001635 +0000 UTC m=+2704.898707469" watchObservedRunningTime="2026-10-30 09:39:47.037662342 +0000 UTC m=+2704.900368162" Oct 30 09:39:49.498837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.498801 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj"] Oct 30 09:39:49.503569 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.503547 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:39:49.521887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.521861 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj"] Oct 30 09:39:49.531657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.531627 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f6f96\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0f6f96\"" Oct 30 09:39:49.531794 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.531632 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f6f96\"/\"routing-overlay\"" Oct 30 09:39:49.531794 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.531643 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f6f96\"/\"openshift-service-ca.crt\"" Oct 30 09:39:49.531997 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.531972 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f6f96\"/\"kube-root-ca.crt\"" Oct 30 09:39:49.542350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.542320 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/155c1886-0a42-4725-9c47-9324d5837ffb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj\" (UID: \"155c1886-0a42-4725-9c47-9324d5837ffb\") " pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:39:49.542561 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.542538 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/155c1886-0a42-4725-9c47-9324d5837ffb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj\" (UID: \"155c1886-0a42-4725-9c47-9324d5837ffb\") " pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:39:49.542652 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.542618 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll4xk\" (UniqueName: \"kubernetes.io/projected/155c1886-0a42-4725-9c47-9324d5837ffb-kube-api-access-ll4xk\") pod \"payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj\" (UID: \"155c1886-0a42-4725-9c47-9324d5837ffb\") " pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:39:49.542763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.542733 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/155c1886-0a42-4725-9c47-9324d5837ffb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj\" (UID: \"155c1886-0a42-4725-9c47-9324d5837ffb\") " pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:39:49.572754 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.572686 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0f6f96\"/\"agc-cred-50cc7c61eb28\"" Oct 30 09:39:49.572907 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.572839 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0f6f96\"/\"payload-processing-external-model-e2e--e9e64a97-dockercfg-sk9lq\"" Oct 30 09:39:49.644610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.644196 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/155c1886-0a42-4725-9c47-9324d5837ffb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj\" (UID: \"155c1886-0a42-4725-9c47-9324d5837ffb\") " pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:39:49.644610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.644253 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll4xk\" (UniqueName: \"kubernetes.io/projected/155c1886-0a42-4725-9c47-9324d5837ffb-kube-api-access-ll4xk\") pod \"payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj\" (UID: \"155c1886-0a42-4725-9c47-9324d5837ffb\") " pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:39:49.644610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.644323 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/155c1886-0a42-4725-9c47-9324d5837ffb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj\" (UID: \"155c1886-0a42-4725-9c47-9324d5837ffb\") " pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:39:49.644610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.644502 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/155c1886-0a42-4725-9c47-9324d5837ffb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj\" (UID: \"155c1886-0a42-4725-9c47-9324d5837ffb\") " pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:39:49.652610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.645386 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/155c1886-0a42-4725-9c47-9324d5837ffb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj\" (UID: \"155c1886-0a42-4725-9c47-9324d5837ffb\") " pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:39:49.652610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.646153 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/155c1886-0a42-4725-9c47-9324d5837ffb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj\" (UID: \"155c1886-0a42-4725-9c47-9324d5837ffb\") " pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:39:49.656186 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.656094 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/155c1886-0a42-4725-9c47-9324d5837ffb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj\" (UID: \"155c1886-0a42-4725-9c47-9324d5837ffb\") " pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:39:49.679016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.678940 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll4xk\" (UniqueName: \"kubernetes.io/projected/155c1886-0a42-4725-9c47-9324d5837ffb-kube-api-access-ll4xk\") pod \"payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj\" (UID: \"155c1886-0a42-4725-9c47-9324d5837ffb\") " pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:39:49.817670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:49.817559 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:39:50.124796 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:50.124769 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj"] Oct 30 09:39:50.126408 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:39:50.126380 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod155c1886_0a42_4725_9c47_9324d5837ffb.slice/crio-cf43b771bd2806f0033534aef3805afc9ba4aa7b2ef39a2f0aef6279a0e4a6d0 WatchSource:0}: Error finding container cf43b771bd2806f0033534aef3805afc9ba4aa7b2ef39a2f0aef6279a0e4a6d0: Status 404 returned error can't find the container with id cf43b771bd2806f0033534aef3805afc9ba4aa7b2ef39a2f0aef6279a0e4a6d0 Oct 30 09:39:51.007431 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:51.007384 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" event={"ID":"155c1886-0a42-4725-9c47-9324d5837ffb","Type":"ContainerStarted","Data":"f209b2797c4c45dbe573fa7378fe86de19386a2a870cd1247e43b8d2cbb49a2b"} Oct 30 09:39:51.007952 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:51.007452 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" event={"ID":"155c1886-0a42-4725-9c47-9324d5837ffb","Type":"ContainerStarted","Data":"cf43b771bd2806f0033534aef3805afc9ba4aa7b2ef39a2f0aef6279a0e4a6d0"} Oct 30 09:39:51.007952 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:51.007481 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:39:51.049690 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:51.049635 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" podStartSLOduration=2.049618069 podStartE2EDuration="2.049618069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:51.048653993 +0000 UTC m=+2708.911359808" watchObservedRunningTime="2026-10-30 09:39:51.049618069 +0000 UTC m=+2708.912323884" Oct 30 09:39:53.035978 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:39:53.035903 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 09:40:02.018180 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:02.018148 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:40:09.488140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:09.488100 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq"] Oct 30 09:40:09.492668 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:09.492644 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:09.512891 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:09.512852 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq"] Oct 30 09:40:09.522416 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:09.522387 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-035d7d-serving-cert\"" Oct 30 09:40:09.534610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:09.534560 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1261fad1-c20a-4074-b4cf-661b40ee5b33-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:09.534809 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:09.534687 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-metrics-certs\") pod \"maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:09.534885 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:09.534829 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-maas-api-tls\") pod \"maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:09.534949 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:09.534917 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k8ft\" (UniqueName: \"kubernetes.io/projected/1261fad1-c20a-4074-b4cf-661b40ee5b33-kube-api-access-5k8ft\") pod \"maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:09.636289 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:09.636254 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8ft\" (UniqueName: \"kubernetes.io/projected/1261fad1-c20a-4074-b4cf-661b40ee5b33-kube-api-access-5k8ft\") pod \"maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:09.636467 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:09.636345 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1261fad1-c20a-4074-b4cf-661b40ee5b33-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:09.636467 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:09.636394 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-metrics-certs\") pod \"maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:09.636552 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:09.636490 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-maas-api-tls\") pod \"maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:09.636698 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:40:09.636677 5938 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-serving-cert: secret "maas-api-e2e-extmod-b-035d7d-serving-cert" not found Oct 30 09:40:09.636769 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:40:09.636759 5938 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-maas-api-tls podName:1261fad1-c20a-4074-b4cf-661b40ee5b33 nodeName:}" failed. No retries permitted until 2026-10-30 09:40:10.136738454 +0000 UTC m=+2727.999444265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-maas-api-tls") pod "maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" (UID: "1261fad1-c20a-4074-b4cf-661b40ee5b33") : secret "maas-api-e2e-extmod-b-035d7d-serving-cert" not found Oct 30 09:40:09.637113 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:09.637088 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1261fad1-c20a-4074-b4cf-661b40ee5b33-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:09.639021 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:09.638997 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-metrics-certs\") pod \"maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:09.656137 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:09.656104 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k8ft\" (UniqueName: \"kubernetes.io/projected/1261fad1-c20a-4074-b4cf-661b40ee5b33-kube-api-access-5k8ft\") pod \"maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:10.143521 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:10.143485 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-maas-api-tls\") pod \"maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:10.146395 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:10.146373 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-maas-api-tls\") pod \"maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:10.408264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:10.408158 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:10.716945 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:10.716915 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq"] Oct 30 09:40:10.718042 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:40:10.718017 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1261fad1_c20a_4074_b4cf_661b40ee5b33.slice/crio-e8c71b93fbeb803cd4fd4b8ad30d1185c486bb037bc7168acb2e8c26fa0baab1 WatchSource:0}: Error finding container e8c71b93fbeb803cd4fd4b8ad30d1185c486bb037bc7168acb2e8c26fa0baab1: Status 404 returned error can't find the container with id e8c71b93fbeb803cd4fd4b8ad30d1185c486bb037bc7168acb2e8c26fa0baab1 Oct 30 09:40:11.157268 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:11.157232 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" event={"ID":"1261fad1-c20a-4074-b4cf-661b40ee5b33","Type":"ContainerStarted","Data":"e8c71b93fbeb803cd4fd4b8ad30d1185c486bb037bc7168acb2e8c26fa0baab1"} Oct 30 09:40:12.165879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:12.165843 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" event={"ID":"1261fad1-c20a-4074-b4cf-661b40ee5b33","Type":"ContainerStarted","Data":"091025fe8e08311613cada8240aa557618b74f34659eacb38ea880260066aa70"} Oct 30 09:40:12.166295 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:12.166023 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:12.208922 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:12.208872 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" podStartSLOduration=2.818467147 podStartE2EDuration="3.208852305s" totalImagesPullingTime="390.385158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:40:12.207831204 +0000 UTC m=+2730.070537017" watchObservedRunningTime="2026-10-30 09:40:12.208852305 +0000 UTC m=+2730.071558125" Oct 30 09:40:15.809489 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:40:15.809449 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:40:18.178448 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:18.178413 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:40:35.336682 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:35.336650 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg"] Oct 30 09:40:35.342395 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:35.342373 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" Oct 30 09:40:35.360459 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:35.360428 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg"] Oct 30 09:40:35.370155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:35.370131 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-f58f32\"" Oct 30 09:40:35.370271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:35.370218 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-f58f32-dockercfg-t9h82\"" Oct 30 09:40:35.449631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:35.448994 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptb6q\" (UniqueName: \"kubernetes.io/projected/bc95bf70-8335-4690-aea8-86934dd0f73f-kube-api-access-ptb6q\") pod \"payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg\" (UID: \"bc95bf70-8335-4690-aea8-86934dd0f73f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" Oct 30 09:40:35.449631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:35.449172 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc95bf70-8335-4690-aea8-86934dd0f73f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg\" (UID: \"bc95bf70-8335-4690-aea8-86934dd0f73f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" Oct 30 09:40:35.550890 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:35.550855 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc95bf70-8335-4690-aea8-86934dd0f73f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg\" (UID: \"bc95bf70-8335-4690-aea8-86934dd0f73f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" Oct 30 09:40:35.551071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:35.551016 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptb6q\" (UniqueName: \"kubernetes.io/projected/bc95bf70-8335-4690-aea8-86934dd0f73f-kube-api-access-ptb6q\") pod \"payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg\" (UID: \"bc95bf70-8335-4690-aea8-86934dd0f73f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" Oct 30 09:40:35.551567 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:35.551542 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc95bf70-8335-4690-aea8-86934dd0f73f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg\" (UID: \"bc95bf70-8335-4690-aea8-86934dd0f73f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" Oct 30 09:40:35.571760 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:35.571727 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptb6q\" (UniqueName: \"kubernetes.io/projected/bc95bf70-8335-4690-aea8-86934dd0f73f-kube-api-access-ptb6q\") pod \"payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg\" (UID: \"bc95bf70-8335-4690-aea8-86934dd0f73f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" Oct 30 09:40:35.656143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:35.656111 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" Oct 30 09:40:35.964034 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:35.964008 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg"] Oct 30 09:40:35.965205 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:40:35.965183 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc95bf70_8335_4690_aea8_86934dd0f73f.slice/crio-5bf4569a47ce1aa15d2d9b2b9607bcf4dcb1a57237f1d597e28c6ac1df458707 WatchSource:0}: Error finding container 5bf4569a47ce1aa15d2d9b2b9607bcf4dcb1a57237f1d597e28c6ac1df458707: Status 404 returned error can't find the container with id 5bf4569a47ce1aa15d2d9b2b9607bcf4dcb1a57237f1d597e28c6ac1df458707 Oct 30 09:40:36.348227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:36.348123 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" event={"ID":"bc95bf70-8335-4690-aea8-86934dd0f73f","Type":"ContainerStarted","Data":"05746cc8740517ee77db67100a307c520056c2f3f0b4434a1051ef0f337d26a4"} Oct 30 09:40:36.348227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:36.348202 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" event={"ID":"bc95bf70-8335-4690-aea8-86934dd0f73f","Type":"ContainerStarted","Data":"5bf4569a47ce1aa15d2d9b2b9607bcf4dcb1a57237f1d597e28c6ac1df458707"} Oct 30 09:40:36.348777 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:36.348304 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" Oct 30 09:40:36.391720 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:36.391670 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" podStartSLOduration=1.39165627 podStartE2EDuration="1.39165627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:40:36.389801231 +0000 UTC m=+2754.252507053" watchObservedRunningTime="2026-10-30 09:40:36.39165627 +0000 UTC m=+2754.254362091" Oct 30 09:40:43.547673 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.547634 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm"] Oct 30 09:40:43.553391 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.553364 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.571076 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.569275 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-6d4b7a\"/\"openshift-service-ca.crt\"" Oct 30 09:40:43.582998 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.582964 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-6d4b7a\"/\"default-dockercfg-rck84\"" Oct 30 09:40:43.583193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.583041 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-6d4b7a\"/\"autores-model-6d4b7a-kserve-self-signed-certs\"" Oct 30 09:40:43.583193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.583082 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-6d4b7a\"/\"kube-root-ca.crt\"" Oct 30 09:40:43.588954 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.588918 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm"] Oct 30 09:40:43.660604 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.660550 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fntcl\" (UniqueName: \"kubernetes.io/projected/f9e8212e-487a-4a9e-93ab-a0a517be36b2-kube-api-access-fntcl\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.660800 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.660712 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-dshm\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.660800 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.660762 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-home\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.660878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.660846 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-tmp-dir\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.660923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.660910 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9e8212e-487a-4a9e-93ab-a0a517be36b2-tls-certs\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.660965 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.660940 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-model-cache\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.762937 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.762880 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fntcl\" (UniqueName: \"kubernetes.io/projected/f9e8212e-487a-4a9e-93ab-a0a517be36b2-kube-api-access-fntcl\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.763157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.762969 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-dshm\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.763157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.763000 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-home\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.763157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.763065 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-tmp-dir\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.763157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.763154 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9e8212e-487a-4a9e-93ab-a0a517be36b2-tls-certs\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.763412 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.763196 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-model-cache\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.764012 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.763759 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-home\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.764012 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.763801 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-model-cache\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.764012 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.763899 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-tmp-dir\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.768609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.766644 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-dshm\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.797613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.795651 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fntcl\" (UniqueName: \"kubernetes.io/projected/f9e8212e-487a-4a9e-93ab-a0a517be36b2-kube-api-access-fntcl\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.840677 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.836828 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9e8212e-487a-4a9e-93ab-a0a517be36b2-tls-certs\") pod \"autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:43.874649 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:43.874610 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:44.186792 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:44.186757 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm"] Oct 30 09:40:44.187713 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:40:44.187683 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9e8212e_487a_4a9e_93ab_a0a517be36b2.slice/crio-172c2405c610dd412ddd6508f8578d9e07b55e051d6f907dc0e86188464296f5 WatchSource:0}: Error finding container 172c2405c610dd412ddd6508f8578d9e07b55e051d6f907dc0e86188464296f5: Status 404 returned error can't find the container with id 172c2405c610dd412ddd6508f8578d9e07b55e051d6f907dc0e86188464296f5 Oct 30 09:40:44.406501 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:44.406468 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" event={"ID":"f9e8212e-487a-4a9e-93ab-a0a517be36b2","Type":"ContainerStarted","Data":"8948a788d22ca47e2601b2612136f78bb93ce4cfc9673a7257dffdcf52bda2a8"} Oct 30 09:40:44.406724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:44.406512 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" event={"ID":"f9e8212e-487a-4a9e-93ab-a0a517be36b2","Type":"ContainerStarted","Data":"172c2405c610dd412ddd6508f8578d9e07b55e051d6f907dc0e86188464296f5"} Oct 30 09:40:44.455913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:44.455798 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" podStartSLOduration=1.455775918 podStartE2EDuration="1.455775918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:40:44.453740712 +0000 UTC m=+2762.316446535" watchObservedRunningTime="2026-10-30 09:40:44.455775918 +0000 UTC m=+2762.318481739" Oct 30 09:40:44.875380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:44.875341 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:45.809925 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:40:45.809879 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:40:47.359034 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:47.359002 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" Oct 30 09:40:53.875095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:53.875056 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:53.888786 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:53.888762 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:40:54.500369 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:40:54.500339 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:41:07.812638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:07.812601 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s"] Oct 30 09:41:07.818411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:07.818393 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" Oct 30 09:41:07.854754 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:07.854713 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:41:07.854928 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:07.854857 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:41:07.860756 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:07.860727 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s"] Oct 30 09:41:07.918271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:07.918233 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c1ae496-hrd5s\" (UID: \"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" Oct 30 09:41:07.918457 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:07.918367 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-676d9\" (UniqueName: \"kubernetes.io/projected/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-kube-api-access-676d9\") pod \"maas-api-revoke-sub-4c1ae496-hrd5s\" (UID: \"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" Oct 30 09:41:07.918457 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:07.918438 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c1ae496-hrd5s\" (UID: \"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" Oct 30 09:41:08.044015 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:08.043973 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c1ae496-hrd5s\" (UID: \"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" Oct 30 09:41:08.044244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:08.044186 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c1ae496-hrd5s\" (UID: \"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" Oct 30 09:41:08.044313 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:08.044271 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676d9\" (UniqueName: \"kubernetes.io/projected/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-kube-api-access-676d9\") pod \"maas-api-revoke-sub-4c1ae496-hrd5s\" (UID: \"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" Oct 30 09:41:08.045403 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:08.045378 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c1ae496-hrd5s\" (UID: \"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" Oct 30 09:41:08.047507 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:08.047481 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c1ae496-hrd5s\" (UID: \"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" Oct 30 09:41:08.065160 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:08.065085 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-676d9\" (UniqueName: \"kubernetes.io/projected/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-kube-api-access-676d9\") pod \"maas-api-revoke-sub-4c1ae496-hrd5s\" (UID: \"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" Oct 30 09:41:08.165817 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:08.165778 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" Oct 30 09:41:08.511405 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:08.511372 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s"] Oct 30 09:41:08.511902 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:41:08.511878 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842dee8f_5b8d_4aca_ae9e_a3f80fe25cd4.slice/crio-8f889bcef10119ed5766c33e5008b4c85581529981bf34336b48f77752a315f8 WatchSource:0}: Error finding container 8f889bcef10119ed5766c33e5008b4c85581529981bf34336b48f77752a315f8: Status 404 returned error can't find the container with id 8f889bcef10119ed5766c33e5008b4c85581529981bf34336b48f77752a315f8 Oct 30 09:41:08.626604 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:08.626557 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" event={"ID":"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4","Type":"ContainerStarted","Data":"8f889bcef10119ed5766c33e5008b4c85581529981bf34336b48f77752a315f8"} Oct 30 09:41:09.636351 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:09.636305 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" event={"ID":"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4","Type":"ContainerStarted","Data":"2df9bb9be9b554800311179f385b93e0063e763f778feaaeec0ffa4d08cca668"} Oct 30 09:41:09.695501 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:09.695449 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" podStartSLOduration=2.69543702 podStartE2EDuration="2.69543702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:41:09.681382144 +0000 UTC m=+2787.544087992" watchObservedRunningTime="2026-10-30 09:41:09.69543702 +0000 UTC m=+2787.558142841" Oct 30 09:41:10.044578 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:10.044473 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm"] Oct 30 09:41:10.651144 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:10.651109 5938 generic.go:356] "Generic (PLEG): container finished" podID="842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4" containerID="2df9bb9be9b554800311179f385b93e0063e763f778feaaeec0ffa4d08cca668" exitCode=0 Oct 30 09:41:10.653157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:10.652203 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" event={"ID":"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4","Type":"ContainerDied","Data":"2df9bb9be9b554800311179f385b93e0063e763f778feaaeec0ffa4d08cca668"} Oct 30 09:41:11.907636 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:11.907579 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" Oct 30 09:41:12.018338 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:12.017973 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-kube-api-access-676d9\" (UniqueName: \"kubernetes.io/projected/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-kube-api-access-676d9\") pod \"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4\" (UID: \"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4\") " Oct 30 09:41:12.018905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:12.018867 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-maas-api-xks-ca\") pod \"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4\" (UID: \"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4\") " Oct 30 09:41:12.019130 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:12.019110 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-maas-api-openshift-ca\") pod \"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4\" (UID: \"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4\") " Oct 30 09:41:12.020408 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:12.020383 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-maas-api-openshift-ca" pod "842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4" (UID: "842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:41:12.048727 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:12.048687 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-kube-api-access-676d9" pod "842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4" (UID: "842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4"). InnerVolumeSpecName "kube-api-access-676d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:41:12.078242 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:12.078205 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-maas-api-xks-ca" pod "842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4" (UID: "842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:41:12.124883 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:12.124739 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:41:12.124883 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:12.124774 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:41:12.124883 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:12.124789 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-676d9\" (UniqueName: \"kubernetes.io/projected/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4-kube-api-access-676d9\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:41:12.673976 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:12.673951 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" Oct 30 09:41:12.674148 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:12.673947 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s" event={"ID":"842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4","Type":"ContainerDied","Data":"8f889bcef10119ed5766c33e5008b4c85581529981bf34336b48f77752a315f8"} Oct 30 09:41:12.674148 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:12.674095 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f889bcef10119ed5766c33e5008b4c85581529981bf34336b48f77752a315f8" Oct 30 09:41:13.684839 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:13.684802 5938 generic.go:356] "Generic (PLEG): container finished" podID="b9985d34-9195-47d7-8e4c-fb5a87ca969a" containerID="062de02c198f95e5e1ae6f5449e1d2dc709521f9184312d820d57dc67dc7fbf9" exitCode=0 Oct 30 09:41:13.685464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:13.684860 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" event={"ID":"b9985d34-9195-47d7-8e4c-fb5a87ca969a","Type":"ContainerDied","Data":"062de02c198f95e5e1ae6f5449e1d2dc709521f9184312d820d57dc67dc7fbf9"} Oct 30 09:41:13.685544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:13.685471 5938 scope.go:122] "RemoveContainer" containerID="062de02c198f95e5e1ae6f5449e1d2dc709521f9184312d820d57dc67dc7fbf9" Oct 30 09:41:13.687689 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:13.687578 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:41:13.687848 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:13.687749 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:41:13.687848 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:13.687797 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:41:14.694453 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:14.694412 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" event={"ID":"b9985d34-9195-47d7-8e4c-fb5a87ca969a","Type":"ContainerStarted","Data":"b149e7d962d7ff3102b9755392e52267b1a27cedd8da786acc266fa14840cb57"} Oct 30 09:41:15.164920 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:15.164884 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:41:15.166130 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:15.166105 5938 patch_prober.go:29] interesting pod/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:41:15.166222 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:15.166156 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" podUID="b9985d34-9195-47d7-8e4c-fb5a87ca969a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 09:41:15.811629 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:41:15.811563 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:41:15.948394 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:15.948356 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:41:16.164750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.164707 5938 patch_prober.go:29] interesting pod/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:41:16.164935 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.164774 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" podUID="b9985d34-9195-47d7-8e4c-fb5a87ca969a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 09:41:16.650341 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.650300 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv"] Oct 30 09:41:16.651077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.651062 5938 cpu_manager.go:414] "containerMap: removing container" podUID="842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4" containerName="revoke-keys" Oct 30 09:41:16.651077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.651079 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4" containerName="revoke-keys" Oct 30 09:41:16.651221 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.651212 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4" containerName="revoke-keys" Oct 30 09:41:16.655316 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.655294 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.673716 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.673685 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv"] Oct 30 09:41:16.682975 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.682939 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-6d4b7a-openshift-default-dockercfg-lnf6h\"" Oct 30 09:41:16.707575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.707523 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-workload-certs\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.707826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.707653 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-envoy\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.707826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.707761 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptn5z\" (UniqueName: \"kubernetes.io/projected/9fc040c6-b1d4-401b-be91-5bd2f0696b98-kube-api-access-ptn5z\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.707941 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.707861 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-credential-socket\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.707941 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.707900 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-data\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.708212 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.708187 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-token\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.708300 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.708230 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-podinfo\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.708381 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.708363 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istiod-ca-cert\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.708440 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.708417 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-workload-socket\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.809457 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.809427 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-workload-certs\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.809685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.809498 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-envoy\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.809685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.809524 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptn5z\" (UniqueName: \"kubernetes.io/projected/9fc040c6-b1d4-401b-be91-5bd2f0696b98-kube-api-access-ptn5z\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.809685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.809558 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-credential-socket\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.809879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.809737 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-data\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.809939 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.809917 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-workload-certs\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.810002 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.809946 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-credential-socket\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.810071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.810020 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-token\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.810071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.810050 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-podinfo\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.810181 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.810134 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-data\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.810234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.810194 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istiod-ca-cert\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.810234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.810219 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-workload-socket\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.810627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.810582 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-workload-socket\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.810870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.810848 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istiod-ca-cert\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.812298 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.812280 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-envoy\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.812699 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.812580 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-podinfo\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.846900 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.846867 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-token\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.847155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.847030 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptn5z\" (UniqueName: \"kubernetes.io/projected/9fc040c6-b1d4-401b-be91-5bd2f0696b98-kube-api-access-ptn5z\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:16.976023 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:16.975929 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:17.165135 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:17.165102 5938 patch_prober.go:29] interesting pod/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:41:17.165344 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:17.165157 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" podUID="b9985d34-9195-47d7-8e4c-fb5a87ca969a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 09:41:17.348891 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:17.348857 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv"] Oct 30 09:41:17.350953 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:41:17.350922 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc040c6_b1d4_401b_be91_5bd2f0696b98.slice/crio-0449de55bd7a34a987675881f74dcc16cce6d62ae6b17e5cde9e9d929a04635d WatchSource:0}: Error finding container 0449de55bd7a34a987675881f74dcc16cce6d62ae6b17e5cde9e9d929a04635d: Status 404 returned error can't find the container with id 0449de55bd7a34a987675881f74dcc16cce6d62ae6b17e5cde9e9d929a04635d Oct 30 09:41:17.354109 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:17.354072 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:41:17.354206 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:17.354145 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:41:17.354206 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:17.354181 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:41:17.750759 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:17.750716 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" event={"ID":"9fc040c6-b1d4-401b-be91-5bd2f0696b98","Type":"ContainerStarted","Data":"b5ec843b0ff099420e4e70ec79b6455d917c6263077d8619b4309cf60ab94332"} Oct 30 09:41:17.751012 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:17.750829 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" event={"ID":"9fc040c6-b1d4-401b-be91-5bd2f0696b98","Type":"ContainerStarted","Data":"0449de55bd7a34a987675881f74dcc16cce6d62ae6b17e5cde9e9d929a04635d"} Oct 30 09:41:17.796677 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:17.796618 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" podStartSLOduration=1.796581007 podStartE2EDuration="1.796581007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:41:17.795185722 +0000 UTC m=+2795.657891559" watchObservedRunningTime="2026-10-30 09:41:17.796581007 +0000 UTC m=+2795.659286832" Oct 30 09:41:17.977043 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:17.977000 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:18.165301 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:18.165268 5938 patch_prober.go:29] interesting pod/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:41:18.165504 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:18.165329 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" podUID="b9985d34-9195-47d7-8e4c-fb5a87ca969a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 09:41:18.978522 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:18.978475 5938 patch_prober.go:29] interesting pod/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 09:41:18.978998 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:18.978563 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" podUID="9fc040c6-b1d4-401b-be91-5bd2f0696b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 09:41:19.085609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:19.085268 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj"] Oct 30 09:41:19.179649 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:19.172229 5938 patch_prober.go:29] interesting pod/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:41:19.190976 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:19.190930 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" podUID="b9985d34-9195-47d7-8e4c-fb5a87ca969a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 09:41:19.977318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:19.977280 5938 patch_prober.go:29] interesting pod/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 09:41:19.977542 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:19.977337 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" podUID="9fc040c6-b1d4-401b-be91-5bd2f0696b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 09:41:20.167078 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:20.167031 5938 patch_prober.go:29] interesting pod/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:41:20.167639 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:20.167100 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" podUID="b9985d34-9195-47d7-8e4c-fb5a87ca969a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 09:41:20.787063 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:20.787030 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq"] Oct 30 09:41:20.977139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:20.977101 5938 patch_prober.go:29] interesting pod/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 09:41:20.977324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:20.977170 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" podUID="9fc040c6-b1d4-401b-be91-5bd2f0696b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 09:41:21.171753 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:21.171717 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:41:21.172949 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:21.172924 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:41:21.976524 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:21.976489 5938 patch_prober.go:29] interesting pod/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 09:41:21.976765 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:21.976562 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" podUID="9fc040c6-b1d4-401b-be91-5bd2f0696b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 09:41:22.976788 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:22.976754 5938 patch_prober.go:29] interesting pod/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 09:41:22.977218 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:22.976818 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" podUID="9fc040c6-b1d4-401b-be91-5bd2f0696b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 09:41:23.976886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:23.976846 5938 patch_prober.go:29] interesting pod/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 09:41:23.977327 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:23.976905 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" podUID="9fc040c6-b1d4-401b-be91-5bd2f0696b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 09:41:24.086523 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.086476 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" podUID="155c1886-0a42-4725-9c47-9324d5837ffb" containerName="payload-processing" containerID="cri-o://f209b2797c4c45dbe573fa7378fe86de19386a2a870cd1247e43b8d2cbb49a2b" gracePeriod=25 Oct 30 09:41:24.391237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.391209 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:41:24.454359 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.454308 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/155c1886-0a42-4725-9c47-9324d5837ffb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/155c1886-0a42-4725-9c47-9324d5837ffb-provider-credentials\") pod \"155c1886-0a42-4725-9c47-9324d5837ffb\" (UID: \"155c1886-0a42-4725-9c47-9324d5837ffb\") " Oct 30 09:41:24.454568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.454405 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/155c1886-0a42-4725-9c47-9324d5837ffb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/155c1886-0a42-4725-9c47-9324d5837ffb-plugins-config-volume\") pod \"155c1886-0a42-4725-9c47-9324d5837ffb\" (UID: \"155c1886-0a42-4725-9c47-9324d5837ffb\") " Oct 30 09:41:24.454568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.454451 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/155c1886-0a42-4725-9c47-9324d5837ffb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/155c1886-0a42-4725-9c47-9324d5837ffb-routing-overlay-volume\") pod \"155c1886-0a42-4725-9c47-9324d5837ffb\" (UID: \"155c1886-0a42-4725-9c47-9324d5837ffb\") " Oct 30 09:41:24.454568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.454519 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/155c1886-0a42-4725-9c47-9324d5837ffb-kube-api-access-ll4xk\" (UniqueName: \"kubernetes.io/projected/155c1886-0a42-4725-9c47-9324d5837ffb-kube-api-access-ll4xk\") pod \"155c1886-0a42-4725-9c47-9324d5837ffb\" (UID: \"155c1886-0a42-4725-9c47-9324d5837ffb\") " Oct 30 09:41:24.455608 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.455543 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155c1886-0a42-4725-9c47-9324d5837ffb-plugins-config-volume" pod "155c1886-0a42-4725-9c47-9324d5837ffb" (UID: "155c1886-0a42-4725-9c47-9324d5837ffb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:41:24.458510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.458480 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155c1886-0a42-4725-9c47-9324d5837ffb-kube-api-access-ll4xk" pod "155c1886-0a42-4725-9c47-9324d5837ffb" (UID: "155c1886-0a42-4725-9c47-9324d5837ffb"). InnerVolumeSpecName "kube-api-access-ll4xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:41:24.458673 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.458609 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155c1886-0a42-4725-9c47-9324d5837ffb-routing-overlay-volume" pod "155c1886-0a42-4725-9c47-9324d5837ffb" (UID: "155c1886-0a42-4725-9c47-9324d5837ffb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:41:24.461148 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.461114 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155c1886-0a42-4725-9c47-9324d5837ffb-provider-credentials" pod "155c1886-0a42-4725-9c47-9324d5837ffb" (UID: "155c1886-0a42-4725-9c47-9324d5837ffb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:41:24.556878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.556796 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll4xk\" (UniqueName: \"kubernetes.io/projected/155c1886-0a42-4725-9c47-9324d5837ffb-kube-api-access-ll4xk\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:41:24.556878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.556825 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/155c1886-0a42-4725-9c47-9324d5837ffb-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:41:24.556878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.556835 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/155c1886-0a42-4725-9c47-9324d5837ffb-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:41:24.556878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.556845 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/155c1886-0a42-4725-9c47-9324d5837ffb-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:41:24.833009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.832906 5938 generic.go:356] "Generic (PLEG): container finished" podID="155c1886-0a42-4725-9c47-9324d5837ffb" containerID="f209b2797c4c45dbe573fa7378fe86de19386a2a870cd1247e43b8d2cbb49a2b" exitCode=0 Oct 30 09:41:24.833009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.832976 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" event={"ID":"155c1886-0a42-4725-9c47-9324d5837ffb","Type":"ContainerDied","Data":"f209b2797c4c45dbe573fa7378fe86de19386a2a870cd1247e43b8d2cbb49a2b"} Oct 30 09:41:24.833274 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.833027 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" event={"ID":"155c1886-0a42-4725-9c47-9324d5837ffb","Type":"ContainerDied","Data":"cf43b771bd2806f0033534aef3805afc9ba4aa7b2ef39a2f0aef6279a0e4a6d0"} Oct 30 09:41:24.833274 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.833056 5938 scope.go:122] "RemoveContainer" containerID="f209b2797c4c45dbe573fa7378fe86de19386a2a870cd1247e43b8d2cbb49a2b" Oct 30 09:41:24.833274 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.833219 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj" Oct 30 09:41:24.854817 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.854791 5938 scope.go:122] "RemoveContainer" containerID="f209b2797c4c45dbe573fa7378fe86de19386a2a870cd1247e43b8d2cbb49a2b" Oct 30 09:41:24.855191 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:41:24.855165 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f209b2797c4c45dbe573fa7378fe86de19386a2a870cd1247e43b8d2cbb49a2b\": container with ID starting with f209b2797c4c45dbe573fa7378fe86de19386a2a870cd1247e43b8d2cbb49a2b not found: ID does not exist" containerID="f209b2797c4c45dbe573fa7378fe86de19386a2a870cd1247e43b8d2cbb49a2b" Oct 30 09:41:24.855293 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.855199 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f209b2797c4c45dbe573fa7378fe86de19386a2a870cd1247e43b8d2cbb49a2b"} err="failed to get container status \"f209b2797c4c45dbe573fa7378fe86de19386a2a870cd1247e43b8d2cbb49a2b\": rpc error: code = NotFound desc = could not find container \"f209b2797c4c45dbe573fa7378fe86de19386a2a870cd1247e43b8d2cbb49a2b\": container with ID starting with f209b2797c4c45dbe573fa7378fe86de19386a2a870cd1247e43b8d2cbb49a2b not found: ID does not exist" Oct 30 09:41:24.898971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.898934 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj"] Oct 30 09:41:24.902654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:24.902619 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0f6f96/payload-processing-external-model-e2e-extmod-b-0f6f96-5755kwmwj"] Oct 30 09:41:25.070817 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.070772 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" podUID="f9e8212e-487a-4a9e-93ab-a0a517be36b2" containerName="main" containerID="cri-o://8948a788d22ca47e2601b2612136f78bb93ce4cfc9673a7257dffdcf52bda2a8" gracePeriod=45 Oct 30 09:41:25.173484 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.173455 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:25.174251 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.174223 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:25.177159 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.177134 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:41:25.354759 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.354730 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:41:25.472071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.472029 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9e8212e-487a-4a9e-93ab-a0a517be36b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9e8212e-487a-4a9e-93ab-a0a517be36b2-tls-certs\") pod \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " Oct 30 09:41:25.472281 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.472145 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-model-cache\") pod \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " Oct 30 09:41:25.472281 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.472250 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9e8212e-487a-4a9e-93ab-a0a517be36b2-kube-api-access-fntcl\" (UniqueName: \"kubernetes.io/projected/f9e8212e-487a-4a9e-93ab-a0a517be36b2-kube-api-access-fntcl\") pod \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " Oct 30 09:41:25.472403 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.472325 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-dshm\") pod \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " Oct 30 09:41:25.472458 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.472434 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-home\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-home\") pod \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " Oct 30 09:41:25.472514 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.472492 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-tmp-dir\") pod \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\" (UID: \"f9e8212e-487a-4a9e-93ab-a0a517be36b2\") " Oct 30 09:41:25.473728 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.473696 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-tmp-dir" pod "f9e8212e-487a-4a9e-93ab-a0a517be36b2" (UID: "f9e8212e-487a-4a9e-93ab-a0a517be36b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:41:25.476790 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.476761 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9e8212e-487a-4a9e-93ab-a0a517be36b2-tls-certs" pod "f9e8212e-487a-4a9e-93ab-a0a517be36b2" (UID: "f9e8212e-487a-4a9e-93ab-a0a517be36b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:41:25.477003 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.476983 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-model-cache" pod "f9e8212e-487a-4a9e-93ab-a0a517be36b2" (UID: "f9e8212e-487a-4a9e-93ab-a0a517be36b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:41:25.479613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.479571 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e8212e-487a-4a9e-93ab-a0a517be36b2-kube-api-access-fntcl" pod "f9e8212e-487a-4a9e-93ab-a0a517be36b2" (UID: "f9e8212e-487a-4a9e-93ab-a0a517be36b2"). InnerVolumeSpecName "kube-api-access-fntcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:41:25.481965 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.481937 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-dshm" pod "f9e8212e-487a-4a9e-93ab-a0a517be36b2" (UID: "f9e8212e-487a-4a9e-93ab-a0a517be36b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:41:25.482191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.482172 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-home" pod "f9e8212e-487a-4a9e-93ab-a0a517be36b2" (UID: "f9e8212e-487a-4a9e-93ab-a0a517be36b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:41:25.574884 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.574849 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:41:25.574884 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.574881 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:41:25.574884 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.574891 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9e8212e-487a-4a9e-93ab-a0a517be36b2-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:41:25.575124 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.574900 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:41:25.575124 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.574909 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fntcl\" (UniqueName: \"kubernetes.io/projected/f9e8212e-487a-4a9e-93ab-a0a517be36b2-kube-api-access-fntcl\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:41:25.575124 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.574918 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9e8212e-487a-4a9e-93ab-a0a517be36b2-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:41:25.787338 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.787292 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" podUID="bc478189-5da9-4f6b-9bdc-c713894f061d" containerName="payload-pre-processing" containerID="cri-o://d583fcdf7165adf7cfb6152f6b39a004f2bd9652db43cc0ce0c2937b12585862" gracePeriod=25 Oct 30 09:41:25.844182 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.844148 5938 generic.go:356] "Generic (PLEG): container finished" podID="f9e8212e-487a-4a9e-93ab-a0a517be36b2" containerID="8948a788d22ca47e2601b2612136f78bb93ce4cfc9673a7257dffdcf52bda2a8" exitCode=0 Oct 30 09:41:25.844386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.844274 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" Oct 30 09:41:25.844386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.844289 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" event={"ID":"f9e8212e-487a-4a9e-93ab-a0a517be36b2","Type":"ContainerDied","Data":"8948a788d22ca47e2601b2612136f78bb93ce4cfc9673a7257dffdcf52bda2a8"} Oct 30 09:41:25.844386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.844348 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm" event={"ID":"f9e8212e-487a-4a9e-93ab-a0a517be36b2","Type":"ContainerDied","Data":"172c2405c610dd412ddd6508f8578d9e07b55e051d6f907dc0e86188464296f5"} Oct 30 09:41:25.844386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.844369 5938 scope.go:122] "RemoveContainer" containerID="8948a788d22ca47e2601b2612136f78bb93ce4cfc9673a7257dffdcf52bda2a8" Oct 30 09:41:25.901015 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.900974 5938 scope.go:122] "RemoveContainer" containerID="8948a788d22ca47e2601b2612136f78bb93ce4cfc9673a7257dffdcf52bda2a8" Oct 30 09:41:25.901456 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:41:25.901430 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8948a788d22ca47e2601b2612136f78bb93ce4cfc9673a7257dffdcf52bda2a8\": container with ID starting with 8948a788d22ca47e2601b2612136f78bb93ce4cfc9673a7257dffdcf52bda2a8 not found: ID does not exist" containerID="8948a788d22ca47e2601b2612136f78bb93ce4cfc9673a7257dffdcf52bda2a8" Oct 30 09:41:25.901572 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.901470 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8948a788d22ca47e2601b2612136f78bb93ce4cfc9673a7257dffdcf52bda2a8"} err="failed to get container status \"8948a788d22ca47e2601b2612136f78bb93ce4cfc9673a7257dffdcf52bda2a8\": rpc error: code = NotFound desc = could not find container \"8948a788d22ca47e2601b2612136f78bb93ce4cfc9673a7257dffdcf52bda2a8\": container with ID starting with 8948a788d22ca47e2601b2612136f78bb93ce4cfc9673a7257dffdcf52bda2a8 not found: ID does not exist" Oct 30 09:41:25.905084 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.905056 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm"] Oct 30 09:41:25.917807 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:25.917780 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-6d4b7a/autores-model-6d4b7a-kserve-5bfbcd4845-wwtcm"] Oct 30 09:41:26.058064 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.058038 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" Oct 30 09:41:26.185967 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.185933 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc478189-5da9-4f6b-9bdc-c713894f061d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc478189-5da9-4f6b-9bdc-c713894f061d-plugins-config-volume\") pod \"bc478189-5da9-4f6b-9bdc-c713894f061d\" (UID: \"bc478189-5da9-4f6b-9bdc-c713894f061d\") " Oct 30 09:41:26.186464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.186008 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc478189-5da9-4f6b-9bdc-c713894f061d-kube-api-access-dsk7h\" (UniqueName: \"kubernetes.io/projected/bc478189-5da9-4f6b-9bdc-c713894f061d-kube-api-access-dsk7h\") pod \"bc478189-5da9-4f6b-9bdc-c713894f061d\" (UID: \"bc478189-5da9-4f6b-9bdc-c713894f061d\") " Oct 30 09:41:26.186464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.186386 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc478189-5da9-4f6b-9bdc-c713894f061d-plugins-config-volume" pod "bc478189-5da9-4f6b-9bdc-c713894f061d" (UID: "bc478189-5da9-4f6b-9bdc-c713894f061d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:41:26.186880 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.186857 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc478189-5da9-4f6b-9bdc-c713894f061d-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:41:26.188787 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.188759 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc478189-5da9-4f6b-9bdc-c713894f061d-kube-api-access-dsk7h" pod "bc478189-5da9-4f6b-9bdc-c713894f061d" (UID: "bc478189-5da9-4f6b-9bdc-c713894f061d"). InnerVolumeSpecName "kube-api-access-dsk7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:41:26.266430 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.266394 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="155c1886-0a42-4725-9c47-9324d5837ffb" path="/var/lib/kubelet/pods/155c1886-0a42-4725-9c47-9324d5837ffb/volumes" Oct 30 09:41:26.266935 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.266920 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9e8212e-487a-4a9e-93ab-a0a517be36b2" path="/var/lib/kubelet/pods/f9e8212e-487a-4a9e-93ab-a0a517be36b2/volumes" Oct 30 09:41:26.288112 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.288085 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsk7h\" (UniqueName: \"kubernetes.io/projected/bc478189-5da9-4f6b-9bdc-c713894f061d-kube-api-access-dsk7h\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:41:26.853294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.853260 5938 generic.go:356] "Generic (PLEG): container finished" podID="bc478189-5da9-4f6b-9bdc-c713894f061d" containerID="d583fcdf7165adf7cfb6152f6b39a004f2bd9652db43cc0ce0c2937b12585862" exitCode=0 Oct 30 09:41:26.853489 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.853309 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" Oct 30 09:41:26.853489 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.853341 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" event={"ID":"bc478189-5da9-4f6b-9bdc-c713894f061d","Type":"ContainerDied","Data":"d583fcdf7165adf7cfb6152f6b39a004f2bd9652db43cc0ce0c2937b12585862"} Oct 30 09:41:26.853489 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.853390 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq" event={"ID":"bc478189-5da9-4f6b-9bdc-c713894f061d","Type":"ContainerDied","Data":"15c871eefef7945b9b8f6b03f8e8ea12eb61abc0d93567370c37b00e77b8fca9"} Oct 30 09:41:26.853489 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.853411 5938 scope.go:122] "RemoveContainer" containerID="d583fcdf7165adf7cfb6152f6b39a004f2bd9652db43cc0ce0c2937b12585862" Oct 30 09:41:26.865380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.865359 5938 scope.go:122] "RemoveContainer" containerID="d583fcdf7165adf7cfb6152f6b39a004f2bd9652db43cc0ce0c2937b12585862" Oct 30 09:41:26.865679 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:41:26.865651 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d583fcdf7165adf7cfb6152f6b39a004f2bd9652db43cc0ce0c2937b12585862\": container with ID starting with d583fcdf7165adf7cfb6152f6b39a004f2bd9652db43cc0ce0c2937b12585862 not found: ID does not exist" containerID="d583fcdf7165adf7cfb6152f6b39a004f2bd9652db43cc0ce0c2937b12585862" Oct 30 09:41:26.865802 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.865683 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d583fcdf7165adf7cfb6152f6b39a004f2bd9652db43cc0ce0c2937b12585862"} err="failed to get container status \"d583fcdf7165adf7cfb6152f6b39a004f2bd9652db43cc0ce0c2937b12585862\": rpc error: code = NotFound desc = could not find container \"d583fcdf7165adf7cfb6152f6b39a004f2bd9652db43cc0ce0c2937b12585862\": container with ID starting with d583fcdf7165adf7cfb6152f6b39a004f2bd9652db43cc0ce0c2937b12585862 not found: ID does not exist" Oct 30 09:41:26.909070 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.909015 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq"] Oct 30 09:41:26.922506 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:26.922475 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6d4b7a-667c9564b7-bkrxq"] Oct 30 09:41:28.257950 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:28.257918 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc478189-5da9-4f6b-9bdc-c713894f061d" path="/var/lib/kubelet/pods/bc478189-5da9-4f6b-9bdc-c713894f061d/volumes" Oct 30 09:41:34.155622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.155568 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl"] Oct 30 09:41:34.156540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.156517 5938 cpu_manager.go:414] "containerMap: removing container" podUID="bc478189-5da9-4f6b-9bdc-c713894f061d" containerName="payload-pre-processing" Oct 30 09:41:34.156540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.156541 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc478189-5da9-4f6b-9bdc-c713894f061d" containerName="payload-pre-processing" Oct 30 09:41:34.156762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.156647 5938 cpu_manager.go:414] "containerMap: removing container" podUID="155c1886-0a42-4725-9c47-9324d5837ffb" containerName="payload-processing" Oct 30 09:41:34.156762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.156659 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="155c1886-0a42-4725-9c47-9324d5837ffb" containerName="payload-processing" Oct 30 09:41:34.156762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.156710 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f9e8212e-487a-4a9e-93ab-a0a517be36b2" containerName="main" Oct 30 09:41:34.156762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.156718 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e8212e-487a-4a9e-93ab-a0a517be36b2" containerName="main" Oct 30 09:41:34.156972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.156918 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e8212e-487a-4a9e-93ab-a0a517be36b2" containerName="main" Oct 30 09:41:34.156972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.156941 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc478189-5da9-4f6b-9bdc-c713894f061d" containerName="payload-pre-processing" Oct 30 09:41:34.156972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.156958 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="155c1886-0a42-4725-9c47-9324d5837ffb" containerName="payload-processing" Oct 30 09:41:34.162569 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.162530 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:41:34.178512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.178476 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl"] Oct 30 09:41:34.187965 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.187926 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ae795b\"/\"routing-overlay\"" Oct 30 09:41:34.187965 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.187941 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ae795b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ae795b\"" Oct 30 09:41:34.187965 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.187945 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ae795b\"/\"openshift-service-ca.crt\"" Oct 30 09:41:34.188279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.188019 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ae795b\"/\"kube-root-ca.crt\"" Oct 30 09:41:34.188279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.188046 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ae795b\"/\"e2e-cmp-legacy-6aa670-provider-cred\"" Oct 30 09:41:34.327039 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.327001 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpcv7\" (UniqueName: \"kubernetes.io/projected/60ad7415-a69c-4663-a924-5aa3a63423fe-kube-api-access-fpcv7\") pod \"payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl\" (UID: \"60ad7415-a69c-4663-a924-5aa3a63423fe\") " pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:41:34.327241 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.327141 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/60ad7415-a69c-4663-a924-5aa3a63423fe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl\" (UID: \"60ad7415-a69c-4663-a924-5aa3a63423fe\") " pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:41:34.327241 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.327204 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ad7415-a69c-4663-a924-5aa3a63423fe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl\" (UID: \"60ad7415-a69c-4663-a924-5aa3a63423fe\") " pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:41:34.327345 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.327328 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/60ad7415-a69c-4663-a924-5aa3a63423fe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl\" (UID: \"60ad7415-a69c-4663-a924-5aa3a63423fe\") " pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:41:34.432090 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.430438 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/60ad7415-a69c-4663-a924-5aa3a63423fe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl\" (UID: \"60ad7415-a69c-4663-a924-5aa3a63423fe\") " pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:41:34.432090 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.430655 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpcv7\" (UniqueName: \"kubernetes.io/projected/60ad7415-a69c-4663-a924-5aa3a63423fe-kube-api-access-fpcv7\") pod \"payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl\" (UID: \"60ad7415-a69c-4663-a924-5aa3a63423fe\") " pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:41:34.432090 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.430757 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/60ad7415-a69c-4663-a924-5aa3a63423fe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl\" (UID: \"60ad7415-a69c-4663-a924-5aa3a63423fe\") " pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:41:34.432090 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.430836 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ad7415-a69c-4663-a924-5aa3a63423fe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl\" (UID: \"60ad7415-a69c-4663-a924-5aa3a63423fe\") " pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:41:34.432090 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.431829 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ad7415-a69c-4663-a924-5aa3a63423fe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl\" (UID: \"60ad7415-a69c-4663-a924-5aa3a63423fe\") " pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:41:34.432520 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.432097 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/60ad7415-a69c-4663-a924-5aa3a63423fe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl\" (UID: \"60ad7415-a69c-4663-a924-5aa3a63423fe\") " pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:41:34.450752 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.450674 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/60ad7415-a69c-4663-a924-5aa3a63423fe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl\" (UID: \"60ad7415-a69c-4663-a924-5aa3a63423fe\") " pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:41:34.474256 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.474214 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpcv7\" (UniqueName: \"kubernetes.io/projected/60ad7415-a69c-4663-a924-5aa3a63423fe-kube-api-access-fpcv7\") pod \"payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl\" (UID: \"60ad7415-a69c-4663-a924-5aa3a63423fe\") " pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:41:34.479098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.479067 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:41:34.801451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.801420 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl"] Oct 30 09:41:34.802276 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:41:34.802253 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ad7415_a69c_4663_a924_5aa3a63423fe.slice/crio-112367a4a772f73733fbcb32a3e200be946d4c01182243dbb83613c30c312b8f WatchSource:0}: Error finding container 112367a4a772f73733fbcb32a3e200be946d4c01182243dbb83613c30c312b8f: Status 404 returned error can't find the container with id 112367a4a772f73733fbcb32a3e200be946d4c01182243dbb83613c30c312b8f Oct 30 09:41:34.935477 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.935428 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" event={"ID":"60ad7415-a69c-4663-a924-5aa3a63423fe","Type":"ContainerStarted","Data":"0898c797bb7d003478c113f638758f32cec3b924ef3f157731b9219fc5319d11"} Oct 30 09:41:34.935679 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.935504 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" event={"ID":"60ad7415-a69c-4663-a924-5aa3a63423fe","Type":"ContainerStarted","Data":"112367a4a772f73733fbcb32a3e200be946d4c01182243dbb83613c30c312b8f"} Oct 30 09:41:34.935679 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.935534 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:41:34.983249 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:34.983193 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" podStartSLOduration=0.983178409 podStartE2EDuration="983.178409ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:41:34.981197087 +0000 UTC m=+2812.843902910" watchObservedRunningTime="2026-10-30 09:41:34.983178409 +0000 UTC m=+2812.845884230" Oct 30 09:41:43.652835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:43.652802 5938 scope.go:122] "RemoveContainer" containerID="db8bdf98a6e5271e11d68850b6569bb34423990fcdd2cc8f5d99f20422be6274" Oct 30 09:41:45.809035 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:41:45.808998 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:41:45.952672 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:45.952641 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:41:54.034739 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:54.034702 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs"] Oct 30 09:41:54.045123 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:54.045091 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214709f1-gn4xs"] Oct 30 09:41:54.266165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:41:54.266121 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e254ef-f6d0-47df-9424-0f7210062ee9" path="/var/lib/kubelet/pods/87e254ef-f6d0-47df-9424-0f7210062ee9/volumes" Oct 30 09:42:04.030455 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:04.030419 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4"] Oct 30 09:42:04.043498 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:04.043467 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0acde49c-xxgc4"] Oct 30 09:42:04.258069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:04.258035 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a61b1d50-c810-4bcb-a406-f884a8f758f9" path="/var/lib/kubelet/pods/a61b1d50-c810-4bcb-a406-f884a8f758f9/volumes" Oct 30 09:42:05.152914 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:05.152868 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv"] Oct 30 09:42:05.153733 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:05.153696 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" podUID="9fc040c6-b1d4-401b-be91-5bd2f0696b98" containerName="istio-proxy" containerID="cri-o://b5ec843b0ff099420e4e70ec79b6455d917c6263077d8619b4309cf60ab94332" gracePeriod=30 Oct 30 09:42:10.176935 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.176579 5938 patch_prober.go:29] interesting pod/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 09:42:10.176935 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.176670 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" podUID="9fc040c6-b1d4-401b-be91-5bd2f0696b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 09:42:10.425402 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.425327 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:42:10.479255 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.479217 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-token\" (UniqueName: \"kubernetes.io/projected/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-token\") pod \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " Oct 30 09:42:10.479443 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.479264 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-credential-socket\") pod \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " Oct 30 09:42:10.479443 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.479372 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-envoy\") pod \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " Oct 30 09:42:10.479535 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.479460 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-podinfo\") pod \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " Oct 30 09:42:10.479535 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.479506 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-workload-certs\") pod \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " Oct 30 09:42:10.479717 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.479576 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-workload-socket\") pod \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " Oct 30 09:42:10.479717 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.479632 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-data\") pod \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " Oct 30 09:42:10.479717 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.479659 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc040c6-b1d4-401b-be91-5bd2f0696b98-kube-api-access-ptn5z\" (UniqueName: \"kubernetes.io/projected/9fc040c6-b1d4-401b-be91-5bd2f0696b98-kube-api-access-ptn5z\") pod \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " Oct 30 09:42:10.479717 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.479676 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-credential-socket" pod "9fc040c6-b1d4-401b-be91-5bd2f0696b98" (UID: "9fc040c6-b1d4-401b-be91-5bd2f0696b98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:42:10.479717 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.479688 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istiod-ca-cert\") pod \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\" (UID: \"9fc040c6-b1d4-401b-be91-5bd2f0696b98\") " Oct 30 09:42:10.480092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.480064 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-workload-certs" pod "9fc040c6-b1d4-401b-be91-5bd2f0696b98" (UID: "9fc040c6-b1d4-401b-be91-5bd2f0696b98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:42:10.480170 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.480152 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-data" pod "9fc040c6-b1d4-401b-be91-5bd2f0696b98" (UID: "9fc040c6-b1d4-401b-be91-5bd2f0696b98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:42:10.480610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.480395 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istiod-ca-cert" pod "9fc040c6-b1d4-401b-be91-5bd2f0696b98" (UID: "9fc040c6-b1d4-401b-be91-5bd2f0696b98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:42:10.480610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.480540 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-workload-socket" pod "9fc040c6-b1d4-401b-be91-5bd2f0696b98" (UID: "9fc040c6-b1d4-401b-be91-5bd2f0696b98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:42:10.480977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.480727 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.480977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.480750 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.480977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.480765 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.480977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.480779 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.481915 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.481890 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-podinfo" pod "9fc040c6-b1d4-401b-be91-5bd2f0696b98" (UID: "9fc040c6-b1d4-401b-be91-5bd2f0696b98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:42:10.481915 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.481909 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-token" pod "9fc040c6-b1d4-401b-be91-5bd2f0696b98" (UID: "9fc040c6-b1d4-401b-be91-5bd2f0696b98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:42:10.482102 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.481929 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-envoy" pod "9fc040c6-b1d4-401b-be91-5bd2f0696b98" (UID: "9fc040c6-b1d4-401b-be91-5bd2f0696b98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:42:10.482403 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.482386 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc040c6-b1d4-401b-be91-5bd2f0696b98-kube-api-access-ptn5z" pod "9fc040c6-b1d4-401b-be91-5bd2f0696b98" (UID: "9fc040c6-b1d4-401b-be91-5bd2f0696b98"). InnerVolumeSpecName "kube-api-access-ptn5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:42:10.582212 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.582176 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.582212 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.582207 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.582212 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.582217 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptn5z\" (UniqueName: \"kubernetes.io/projected/9fc040c6-b1d4-401b-be91-5bd2f0696b98-kube-api-access-ptn5z\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.582463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.582227 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.582463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:10.582237 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc040c6-b1d4-401b-be91-5bd2f0696b98-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:42:11.245818 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:11.245779 5938 generic.go:356] "Generic (PLEG): container finished" podID="9fc040c6-b1d4-401b-be91-5bd2f0696b98" containerID="b5ec843b0ff099420e4e70ec79b6455d917c6263077d8619b4309cf60ab94332" exitCode=0 Oct 30 09:42:11.246264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:11.245829 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" Oct 30 09:42:11.246264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:11.245862 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" event={"ID":"9fc040c6-b1d4-401b-be91-5bd2f0696b98","Type":"ContainerDied","Data":"b5ec843b0ff099420e4e70ec79b6455d917c6263077d8619b4309cf60ab94332"} Oct 30 09:42:11.246264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:11.245924 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv" event={"ID":"9fc040c6-b1d4-401b-be91-5bd2f0696b98","Type":"ContainerDied","Data":"0449de55bd7a34a987675881f74dcc16cce6d62ae6b17e5cde9e9d929a04635d"} Oct 30 09:42:11.246264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:11.245954 5938 scope.go:122] "RemoveContainer" containerID="b5ec843b0ff099420e4e70ec79b6455d917c6263077d8619b4309cf60ab94332" Oct 30 09:42:11.257732 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:11.257713 5938 scope.go:122] "RemoveContainer" containerID="b5ec843b0ff099420e4e70ec79b6455d917c6263077d8619b4309cf60ab94332" Oct 30 09:42:11.258046 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:42:11.258026 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ec843b0ff099420e4e70ec79b6455d917c6263077d8619b4309cf60ab94332\": container with ID starting with b5ec843b0ff099420e4e70ec79b6455d917c6263077d8619b4309cf60ab94332 not found: ID does not exist" containerID="b5ec843b0ff099420e4e70ec79b6455d917c6263077d8619b4309cf60ab94332" Oct 30 09:42:11.258095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:11.258056 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ec843b0ff099420e4e70ec79b6455d917c6263077d8619b4309cf60ab94332"} err="failed to get container status \"b5ec843b0ff099420e4e70ec79b6455d917c6263077d8619b4309cf60ab94332\": rpc error: code = NotFound desc = could not find container \"b5ec843b0ff099420e4e70ec79b6455d917c6263077d8619b4309cf60ab94332\": container with ID starting with b5ec843b0ff099420e4e70ec79b6455d917c6263077d8619b4309cf60ab94332 not found: ID does not exist" Oct 30 09:42:11.316531 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:11.316484 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv"] Oct 30 09:42:11.330481 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:11.330447 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-9cnhv"] Oct 30 09:42:12.258783 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:12.258746 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc040c6-b1d4-401b-be91-5bd2f0696b98" path="/var/lib/kubelet/pods/9fc040c6-b1d4-401b-be91-5bd2f0696b98/volumes" Oct 30 09:42:15.809229 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:42:15.809193 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:42:19.331082 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.331037 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r"] Oct 30 09:42:19.332154 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.332128 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9fc040c6-b1d4-401b-be91-5bd2f0696b98" containerName="istio-proxy" Oct 30 09:42:19.332154 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.332154 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc040c6-b1d4-401b-be91-5bd2f0696b98" containerName="istio-proxy" Oct 30 09:42:19.332345 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.332296 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc040c6-b1d4-401b-be91-5bd2f0696b98" containerName="istio-proxy" Oct 30 09:42:19.338067 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.338041 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r" Oct 30 09:42:19.349186 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.349152 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r"] Oct 30 09:42:19.355725 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.355698 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:42:19.355986 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.355967 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:42:19.527439 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.527399 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjwzw\" (UniqueName: \"kubernetes.io/projected/3dea5010-8500-425d-832a-e582e6258d00-kube-api-access-pjwzw\") pod \"maas-api-revoke-sub-32eede41-djn2r\" (UID: \"3dea5010-8500-425d-832a-e582e6258d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r" Oct 30 09:42:19.527661 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.527465 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dea5010-8500-425d-832a-e582e6258d00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32eede41-djn2r\" (UID: \"3dea5010-8500-425d-832a-e582e6258d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r" Oct 30 09:42:19.527905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.527879 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dea5010-8500-425d-832a-e582e6258d00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32eede41-djn2r\" (UID: \"3dea5010-8500-425d-832a-e582e6258d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r" Oct 30 09:42:19.628893 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.628855 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dea5010-8500-425d-832a-e582e6258d00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32eede41-djn2r\" (UID: \"3dea5010-8500-425d-832a-e582e6258d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r" Oct 30 09:42:19.629130 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.628928 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjwzw\" (UniqueName: \"kubernetes.io/projected/3dea5010-8500-425d-832a-e582e6258d00-kube-api-access-pjwzw\") pod \"maas-api-revoke-sub-32eede41-djn2r\" (UID: \"3dea5010-8500-425d-832a-e582e6258d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r" Oct 30 09:42:19.629130 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.628964 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dea5010-8500-425d-832a-e582e6258d00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32eede41-djn2r\" (UID: \"3dea5010-8500-425d-832a-e582e6258d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r" Oct 30 09:42:19.629634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.629575 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dea5010-8500-425d-832a-e582e6258d00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32eede41-djn2r\" (UID: \"3dea5010-8500-425d-832a-e582e6258d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r" Oct 30 09:42:19.631563 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.631541 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dea5010-8500-425d-832a-e582e6258d00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32eede41-djn2r\" (UID: \"3dea5010-8500-425d-832a-e582e6258d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r" Oct 30 09:42:19.649685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.649650 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjwzw\" (UniqueName: \"kubernetes.io/projected/3dea5010-8500-425d-832a-e582e6258d00-kube-api-access-pjwzw\") pod \"maas-api-revoke-sub-32eede41-djn2r\" (UID: \"3dea5010-8500-425d-832a-e582e6258d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r" Oct 30 09:42:19.658168 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.658134 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r" Oct 30 09:42:19.967505 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:42:19.967464 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dea5010_8500_425d_832a_e582e6258d00.slice/crio-468580c0bc2d1eecf277303a862fbea87de7b440b746130bda9576cd0bed410e WatchSource:0}: Error finding container 468580c0bc2d1eecf277303a862fbea87de7b440b746130bda9576cd0bed410e: Status 404 returned error can't find the container with id 468580c0bc2d1eecf277303a862fbea87de7b440b746130bda9576cd0bed410e Oct 30 09:42:19.967505 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:19.967486 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r"] Oct 30 09:42:20.330091 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:20.330056 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r" event={"ID":"3dea5010-8500-425d-832a-e582e6258d00","Type":"ContainerStarted","Data":"468580c0bc2d1eecf277303a862fbea87de7b440b746130bda9576cd0bed410e"} Oct 30 09:42:21.341006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:21.340902 5938 generic.go:356] "Generic (PLEG): container finished" podID="3dea5010-8500-425d-832a-e582e6258d00" containerID="83b4c6b1b762e42611fe8c204a24934203f5d7fe439a63bd1c3356f8404238bd" exitCode=0 Oct 30 09:42:21.341006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:21.340990 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r" event={"ID":"3dea5010-8500-425d-832a-e582e6258d00","Type":"ContainerDied","Data":"83b4c6b1b762e42611fe8c204a24934203f5d7fe439a63bd1c3356f8404238bd"} Oct 30 09:42:22.514323 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:22.514301 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r" Oct 30 09:42:22.573809 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:22.573775 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dea5010-8500-425d-832a-e582e6258d00-kube-api-access-pjwzw\" (UniqueName: \"kubernetes.io/projected/3dea5010-8500-425d-832a-e582e6258d00-kube-api-access-pjwzw\") pod \"3dea5010-8500-425d-832a-e582e6258d00\" (UID: \"3dea5010-8500-425d-832a-e582e6258d00\") " Oct 30 09:42:22.574070 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:22.574047 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dea5010-8500-425d-832a-e582e6258d00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dea5010-8500-425d-832a-e582e6258d00-maas-api-openshift-ca\") pod \"3dea5010-8500-425d-832a-e582e6258d00\" (UID: \"3dea5010-8500-425d-832a-e582e6258d00\") " Oct 30 09:42:22.574163 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:22.574149 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dea5010-8500-425d-832a-e582e6258d00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dea5010-8500-425d-832a-e582e6258d00-maas-api-xks-ca\") pod \"3dea5010-8500-425d-832a-e582e6258d00\" (UID: \"3dea5010-8500-425d-832a-e582e6258d00\") " Oct 30 09:42:22.574540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:22.574512 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dea5010-8500-425d-832a-e582e6258d00-maas-api-openshift-ca" pod "3dea5010-8500-425d-832a-e582e6258d00" (UID: "3dea5010-8500-425d-832a-e582e6258d00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:42:22.574789 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:22.574768 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dea5010-8500-425d-832a-e582e6258d00-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:42:22.577008 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:22.576985 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dea5010-8500-425d-832a-e582e6258d00-maas-api-xks-ca" pod "3dea5010-8500-425d-832a-e582e6258d00" (UID: "3dea5010-8500-425d-832a-e582e6258d00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:42:22.577281 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:22.577263 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dea5010-8500-425d-832a-e582e6258d00-kube-api-access-pjwzw" pod "3dea5010-8500-425d-832a-e582e6258d00" (UID: "3dea5010-8500-425d-832a-e582e6258d00"). InnerVolumeSpecName "kube-api-access-pjwzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:42:22.676391 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:22.676359 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dea5010-8500-425d-832a-e582e6258d00-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:42:22.676391 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:22.676388 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjwzw\" (UniqueName: \"kubernetes.io/projected/3dea5010-8500-425d-832a-e582e6258d00-kube-api-access-pjwzw\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:42:23.357336 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:23.357302 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r" event={"ID":"3dea5010-8500-425d-832a-e582e6258d00","Type":"ContainerDied","Data":"468580c0bc2d1eecf277303a862fbea87de7b440b746130bda9576cd0bed410e"} Oct 30 09:42:23.357336 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:23.357331 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r" Oct 30 09:42:23.357627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:23.357356 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="468580c0bc2d1eecf277303a862fbea87de7b440b746130bda9576cd0bed410e" Oct 30 09:42:24.126535 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.126497 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t"] Oct 30 09:42:24.127325 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.127299 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3dea5010-8500-425d-832a-e582e6258d00" containerName="revoke-keys" Oct 30 09:42:24.127325 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.127323 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dea5010-8500-425d-832a-e582e6258d00" containerName="revoke-keys" Oct 30 09:42:24.127559 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.127499 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dea5010-8500-425d-832a-e582e6258d00" containerName="revoke-keys" Oct 30 09:42:24.131337 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.131316 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" Oct 30 09:42:24.149753 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.149725 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t"] Oct 30 09:42:24.159001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.158975 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-88cbc7\"" Oct 30 09:42:24.261199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.261151 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-88cbc7-dockercfg-ggx8b\"" Oct 30 09:42:24.282029 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.281994 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx7lg\" (UniqueName: \"kubernetes.io/projected/2283ff76-dc88-4578-a61a-5dc1be3887e3-kube-api-access-fx7lg\") pod \"payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t\" (UID: \"2283ff76-dc88-4578-a61a-5dc1be3887e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" Oct 30 09:42:24.296045 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.296014 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2283ff76-dc88-4578-a61a-5dc1be3887e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t\" (UID: \"2283ff76-dc88-4578-a61a-5dc1be3887e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" Oct 30 09:42:24.296559 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.292996 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx"] Oct 30 09:42:24.306917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.306886 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx"] Oct 30 09:42:24.307302 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.307229 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" Oct 30 09:42:24.398083 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.397982 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f99edfef-7092-41b9-9d12-dd378b3fa482-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx\" (UID: \"f99edfef-7092-41b9-9d12-dd378b3fa482\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" Oct 30 09:42:24.398658 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.398383 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7lg\" (UniqueName: \"kubernetes.io/projected/2283ff76-dc88-4578-a61a-5dc1be3887e3-kube-api-access-fx7lg\") pod \"payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t\" (UID: \"2283ff76-dc88-4578-a61a-5dc1be3887e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" Oct 30 09:42:24.398809 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.398700 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2283ff76-dc88-4578-a61a-5dc1be3887e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t\" (UID: \"2283ff76-dc88-4578-a61a-5dc1be3887e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" Oct 30 09:42:24.398809 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.398789 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zflsm\" (UniqueName: \"kubernetes.io/projected/f99edfef-7092-41b9-9d12-dd378b3fa482-kube-api-access-zflsm\") pod \"payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx\" (UID: \"f99edfef-7092-41b9-9d12-dd378b3fa482\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" Oct 30 09:42:24.399484 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.399465 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2283ff76-dc88-4578-a61a-5dc1be3887e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t\" (UID: \"2283ff76-dc88-4578-a61a-5dc1be3887e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" Oct 30 09:42:24.420435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.420397 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx7lg\" (UniqueName: \"kubernetes.io/projected/2283ff76-dc88-4578-a61a-5dc1be3887e3-kube-api-access-fx7lg\") pod \"payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t\" (UID: \"2283ff76-dc88-4578-a61a-5dc1be3887e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" Oct 30 09:42:24.447657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.447624 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" Oct 30 09:42:24.499765 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.499718 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f99edfef-7092-41b9-9d12-dd378b3fa482-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx\" (UID: \"f99edfef-7092-41b9-9d12-dd378b3fa482\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" Oct 30 09:42:24.499950 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.499926 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zflsm\" (UniqueName: \"kubernetes.io/projected/f99edfef-7092-41b9-9d12-dd378b3fa482-kube-api-access-zflsm\") pod \"payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx\" (UID: \"f99edfef-7092-41b9-9d12-dd378b3fa482\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" Oct 30 09:42:24.500807 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.500782 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f99edfef-7092-41b9-9d12-dd378b3fa482-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx\" (UID: \"f99edfef-7092-41b9-9d12-dd378b3fa482\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" Oct 30 09:42:24.522714 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.522647 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zflsm\" (UniqueName: \"kubernetes.io/projected/f99edfef-7092-41b9-9d12-dd378b3fa482-kube-api-access-zflsm\") pod \"payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx\" (UID: \"f99edfef-7092-41b9-9d12-dd378b3fa482\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" Oct 30 09:42:24.626050 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.626008 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" Oct 30 09:42:24.756949 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.756918 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t"] Oct 30 09:42:24.759201 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:42:24.759171 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2283ff76_dc88_4578_a61a_5dc1be3887e3.slice/crio-758f17854f6b8c9a0c705703c0c63299ae2f8330f3bce8a112955c71193834c8 WatchSource:0}: Error finding container 758f17854f6b8c9a0c705703c0c63299ae2f8330f3bce8a112955c71193834c8: Status 404 returned error can't find the container with id 758f17854f6b8c9a0c705703c0c63299ae2f8330f3bce8a112955c71193834c8 Oct 30 09:42:24.935948 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:24.935916 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx"] Oct 30 09:42:24.936449 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:42:24.936425 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99edfef_7092_41b9_9d12_dd378b3fa482.slice/crio-1c4d8f539dc3d7846669c2fa4784645fe269b2cd9d1df765ea0aa88d2be004b4 WatchSource:0}: Error finding container 1c4d8f539dc3d7846669c2fa4784645fe269b2cd9d1df765ea0aa88d2be004b4: Status 404 returned error can't find the container with id 1c4d8f539dc3d7846669c2fa4784645fe269b2cd9d1df765ea0aa88d2be004b4 Oct 30 09:42:25.373205 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:25.373168 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" event={"ID":"2283ff76-dc88-4578-a61a-5dc1be3887e3","Type":"ContainerStarted","Data":"b47ac3506e92f9f71e375437f9756fe09117edb89df0e66bf1b8d4182f326bc6"} Oct 30 09:42:25.373805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:25.373234 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" event={"ID":"2283ff76-dc88-4578-a61a-5dc1be3887e3","Type":"ContainerStarted","Data":"758f17854f6b8c9a0c705703c0c63299ae2f8330f3bce8a112955c71193834c8"} Oct 30 09:42:25.373805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:25.373438 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" Oct 30 09:42:25.375286 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:25.375255 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" event={"ID":"f99edfef-7092-41b9-9d12-dd378b3fa482","Type":"ContainerStarted","Data":"0389144b2fd8df0861ff8a798b7dad05059a1fdb7b121cb9d81a0bcc6fcba67d"} Oct 30 09:42:25.375438 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:25.375307 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" event={"ID":"f99edfef-7092-41b9-9d12-dd378b3fa482","Type":"ContainerStarted","Data":"1c4d8f539dc3d7846669c2fa4784645fe269b2cd9d1df765ea0aa88d2be004b4"} Oct 30 09:42:25.375438 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:25.375342 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" Oct 30 09:42:25.417087 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:25.417032 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" podStartSLOduration=1.417018048 podStartE2EDuration="1.417018048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:42:25.413690047 +0000 UTC m=+2863.276395868" watchObservedRunningTime="2026-10-30 09:42:25.417018048 +0000 UTC m=+2863.279723900" Oct 30 09:42:25.454881 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:25.454818 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" podStartSLOduration=1.454799288 podStartE2EDuration="1.454799288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:42:25.453555354 +0000 UTC m=+2863.316261175" watchObservedRunningTime="2026-10-30 09:42:25.454799288 +0000 UTC m=+2863.317505109" Oct 30 09:42:36.386669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:36.386636 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" Oct 30 09:42:36.387172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:36.386717 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" Oct 30 09:42:45.811060 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:42:45.811018 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:42:46.711239 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.711199 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c"] Oct 30 09:42:46.717575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.717544 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:42:46.733952 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.733920 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"agc-cred-cf9bd9a0dc6b\"" Oct 30 09:42:46.734701 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.734679 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c"] Oct 30 09:42:46.744541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.744516 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"kube-root-ca.crt\"" Oct 30 09:42:46.744734 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.744712 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"routing-overlay\"" Oct 30 09:42:46.744846 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.744712 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"openshift-service-ca.crt\"" Oct 30 09:42:46.744846 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.744672 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"payload-processing-external-model-e2e--d06f4d1b-dockercfg-nr74x\"" Oct 30 09:42:46.744846 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.744798 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"payload-processing-external-model-plugins-e2e-extmod-a-881f83\"" Oct 30 09:42:46.850320 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.850270 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6cf721-374c-4a4e-b31f-17a351c4adf3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c\" (UID: \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:42:46.850887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.850447 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd6cf721-374c-4a4e-b31f-17a351c4adf3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c\" (UID: \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:42:46.850887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.850568 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv6xc\" (UniqueName: \"kubernetes.io/projected/bd6cf721-374c-4a4e-b31f-17a351c4adf3-kube-api-access-zv6xc\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c\" (UID: \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:42:46.850887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.850811 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd6cf721-374c-4a4e-b31f-17a351c4adf3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c\" (UID: \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:42:46.952604 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.952542 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd6cf721-374c-4a4e-b31f-17a351c4adf3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c\" (UID: \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:42:46.952830 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.952718 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6cf721-374c-4a4e-b31f-17a351c4adf3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c\" (UID: \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:42:46.952830 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.952795 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd6cf721-374c-4a4e-b31f-17a351c4adf3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c\" (UID: \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:42:46.952953 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.952900 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv6xc\" (UniqueName: \"kubernetes.io/projected/bd6cf721-374c-4a4e-b31f-17a351c4adf3-kube-api-access-zv6xc\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c\" (UID: \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:42:46.953388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.953360 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd6cf721-374c-4a4e-b31f-17a351c4adf3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c\" (UID: \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:42:46.953573 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.953554 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6cf721-374c-4a4e-b31f-17a351c4adf3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c\" (UID: \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:42:46.955803 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.955778 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd6cf721-374c-4a4e-b31f-17a351c4adf3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c\" (UID: \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:42:46.975883 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:46.975809 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv6xc\" (UniqueName: \"kubernetes.io/projected/bd6cf721-374c-4a4e-b31f-17a351c4adf3-kube-api-access-zv6xc\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c\" (UID: \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:42:47.032029 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:47.031986 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:42:47.353766 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:42:47.353732 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6cf721_374c_4a4e_b31f_17a351c4adf3.slice/crio-cc479c356568b14ba48f1891fcdcc0245a01610636a1863a37afda0ab8feedec WatchSource:0}: Error finding container cc479c356568b14ba48f1891fcdcc0245a01610636a1863a37afda0ab8feedec: Status 404 returned error can't find the container with id cc479c356568b14ba48f1891fcdcc0245a01610636a1863a37afda0ab8feedec Oct 30 09:42:47.354633 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:47.354607 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c"] Oct 30 09:42:47.554832 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:47.554790 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" event={"ID":"bd6cf721-374c-4a4e-b31f-17a351c4adf3","Type":"ContainerStarted","Data":"742a8e6935792d5e4a8aa63b2648511ba5d8eeb81a615a1378db574fb96cd4e7"} Oct 30 09:42:47.555010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:47.554860 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" event={"ID":"bd6cf721-374c-4a4e-b31f-17a351c4adf3","Type":"ContainerStarted","Data":"cc479c356568b14ba48f1891fcdcc0245a01610636a1863a37afda0ab8feedec"} Oct 30 09:42:47.555010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:47.554886 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:42:47.598099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:47.598038 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" podStartSLOduration=1.598017835 podStartE2EDuration="1.598017835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:42:47.594948947 +0000 UTC m=+2885.457654769" watchObservedRunningTime="2026-10-30 09:42:47.598017835 +0000 UTC m=+2885.460723655" Oct 30 09:42:53.028144 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:53.028044 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk"] Oct 30 09:42:53.041840 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:53.041805 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab17a479-lvzvk"] Oct 30 09:42:54.260430 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:54.260391 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15d9dfaf-928b-4ffe-9a11-4d859f755710" path="/var/lib/kubelet/pods/15d9dfaf-928b-4ffe-9a11-4d859f755710/volumes" Oct 30 09:42:55.053688 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:55.053654 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4"] Oct 30 09:42:55.069000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:55.068967 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfb90b3a-ckhj4"] Oct 30 09:42:56.259131 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:56.259088 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee04d9d8-c7b3-409b-bbcd-b7137d417cf2" path="/var/lib/kubelet/pods/ee04d9d8-c7b3-409b-bbcd-b7137d417cf2/volumes" Oct 30 09:42:58.565128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:42:58.565086 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:43:01.915957 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:01.915915 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn"] Oct 30 09:43:01.922370 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:01.922334 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn" Oct 30 09:43:01.939218 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:01.939185 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn"] Oct 30 09:43:01.948631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:01.948555 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:43:01.948859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:01.948716 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:43:02.000697 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:02.000655 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5563d9a-edff-4a2e-9dcd-001e876851f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ec72d3-5zsvn\" (UID: \"d5563d9a-edff-4a2e-9dcd-001e876851f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn" Oct 30 09:43:02.000906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:02.000710 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbtwj\" (UniqueName: \"kubernetes.io/projected/d5563d9a-edff-4a2e-9dcd-001e876851f8-kube-api-access-xbtwj\") pod \"maas-api-revoke-sub-40ec72d3-5zsvn\" (UID: \"d5563d9a-edff-4a2e-9dcd-001e876851f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn" Oct 30 09:43:02.000966 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:02.000946 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5563d9a-edff-4a2e-9dcd-001e876851f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ec72d3-5zsvn\" (UID: \"d5563d9a-edff-4a2e-9dcd-001e876851f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn" Oct 30 09:43:02.102792 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:02.102753 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5563d9a-edff-4a2e-9dcd-001e876851f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ec72d3-5zsvn\" (UID: \"d5563d9a-edff-4a2e-9dcd-001e876851f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn" Oct 30 09:43:02.102792 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:02.102807 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbtwj\" (UniqueName: \"kubernetes.io/projected/d5563d9a-edff-4a2e-9dcd-001e876851f8-kube-api-access-xbtwj\") pod \"maas-api-revoke-sub-40ec72d3-5zsvn\" (UID: \"d5563d9a-edff-4a2e-9dcd-001e876851f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn" Oct 30 09:43:02.103090 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:02.102954 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5563d9a-edff-4a2e-9dcd-001e876851f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ec72d3-5zsvn\" (UID: \"d5563d9a-edff-4a2e-9dcd-001e876851f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn" Oct 30 09:43:02.103680 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:02.103654 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5563d9a-edff-4a2e-9dcd-001e876851f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ec72d3-5zsvn\" (UID: \"d5563d9a-edff-4a2e-9dcd-001e876851f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn" Oct 30 09:43:02.105634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:02.105572 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5563d9a-edff-4a2e-9dcd-001e876851f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ec72d3-5zsvn\" (UID: \"d5563d9a-edff-4a2e-9dcd-001e876851f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn" Oct 30 09:43:02.123842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:02.123812 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbtwj\" (UniqueName: \"kubernetes.io/projected/d5563d9a-edff-4a2e-9dcd-001e876851f8-kube-api-access-xbtwj\") pod \"maas-api-revoke-sub-40ec72d3-5zsvn\" (UID: \"d5563d9a-edff-4a2e-9dcd-001e876851f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn" Oct 30 09:43:02.240744 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:02.240626 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn" Oct 30 09:43:02.568714 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:02.568683 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn"] Oct 30 09:43:02.570875 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:43:02.570837 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5563d9a_edff_4a2e_9dcd_001e876851f8.slice/crio-a8cdd3ebcafb65ccae8a77097e5a3b6fb9bd6252e694afdc86cda430acd4b114 WatchSource:0}: Error finding container a8cdd3ebcafb65ccae8a77097e5a3b6fb9bd6252e694afdc86cda430acd4b114: Status 404 returned error can't find the container with id a8cdd3ebcafb65ccae8a77097e5a3b6fb9bd6252e694afdc86cda430acd4b114 Oct 30 09:43:02.689020 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:02.688976 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn" event={"ID":"d5563d9a-edff-4a2e-9dcd-001e876851f8","Type":"ContainerStarted","Data":"a8cdd3ebcafb65ccae8a77097e5a3b6fb9bd6252e694afdc86cda430acd4b114"} Oct 30 09:43:03.700029 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:03.699994 5938 generic.go:356] "Generic (PLEG): container finished" podID="d5563d9a-edff-4a2e-9dcd-001e876851f8" containerID="44b0ba5e14b69a4e8df559a23f03c1b54f2ab61fc76f4e07a11788856d00e44f" exitCode=0 Oct 30 09:43:03.700704 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:03.700073 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn" event={"ID":"d5563d9a-edff-4a2e-9dcd-001e876851f8","Type":"ContainerDied","Data":"44b0ba5e14b69a4e8df559a23f03c1b54f2ab61fc76f4e07a11788856d00e44f"} Oct 30 09:43:04.028738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:04.028700 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l"] Oct 30 09:43:04.032054 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:04.032023 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-882b02de-zld9l"] Oct 30 09:43:04.259559 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:04.259515 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb11947-f2ef-4a23-880e-17f2cccf7c0e" path="/var/lib/kubelet/pods/6cb11947-f2ef-4a23-880e-17f2cccf7c0e/volumes" Oct 30 09:43:04.868282 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:04.868255 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn" Oct 30 09:43:04.945832 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:04.945784 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5563d9a-edff-4a2e-9dcd-001e876851f8-kube-api-access-xbtwj\" (UniqueName: \"kubernetes.io/projected/d5563d9a-edff-4a2e-9dcd-001e876851f8-kube-api-access-xbtwj\") pod \"d5563d9a-edff-4a2e-9dcd-001e876851f8\" (UID: \"d5563d9a-edff-4a2e-9dcd-001e876851f8\") " Oct 30 09:43:04.946045 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:04.946027 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5563d9a-edff-4a2e-9dcd-001e876851f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5563d9a-edff-4a2e-9dcd-001e876851f8-maas-api-openshift-ca\") pod \"d5563d9a-edff-4a2e-9dcd-001e876851f8\" (UID: \"d5563d9a-edff-4a2e-9dcd-001e876851f8\") " Oct 30 09:43:04.946157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:04.946141 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5563d9a-edff-4a2e-9dcd-001e876851f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5563d9a-edff-4a2e-9dcd-001e876851f8-maas-api-xks-ca\") pod \"d5563d9a-edff-4a2e-9dcd-001e876851f8\" (UID: \"d5563d9a-edff-4a2e-9dcd-001e876851f8\") " Oct 30 09:43:04.946424 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:04.946398 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5563d9a-edff-4a2e-9dcd-001e876851f8-maas-api-openshift-ca" pod "d5563d9a-edff-4a2e-9dcd-001e876851f8" (UID: "d5563d9a-edff-4a2e-9dcd-001e876851f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:43:04.947108 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:04.947079 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5563d9a-edff-4a2e-9dcd-001e876851f8-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:04.948751 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:04.948721 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5563d9a-edff-4a2e-9dcd-001e876851f8-maas-api-xks-ca" pod "d5563d9a-edff-4a2e-9dcd-001e876851f8" (UID: "d5563d9a-edff-4a2e-9dcd-001e876851f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:43:04.951197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:04.951170 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5563d9a-edff-4a2e-9dcd-001e876851f8-kube-api-access-xbtwj" pod "d5563d9a-edff-4a2e-9dcd-001e876851f8" (UID: "d5563d9a-edff-4a2e-9dcd-001e876851f8"). InnerVolumeSpecName "kube-api-access-xbtwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:43:05.048507 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:05.048464 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5563d9a-edff-4a2e-9dcd-001e876851f8-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:05.048507 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:05.048503 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbtwj\" (UniqueName: \"kubernetes.io/projected/d5563d9a-edff-4a2e-9dcd-001e876851f8-kube-api-access-xbtwj\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:05.715423 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:05.715384 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn" event={"ID":"d5563d9a-edff-4a2e-9dcd-001e876851f8","Type":"ContainerDied","Data":"a8cdd3ebcafb65ccae8a77097e5a3b6fb9bd6252e694afdc86cda430acd4b114"} Oct 30 09:43:05.715423 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:05.715421 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn" Oct 30 09:43:05.715738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:05.715440 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8cdd3ebcafb65ccae8a77097e5a3b6fb9bd6252e694afdc86cda430acd4b114" Oct 30 09:43:06.027710 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:06.027612 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852"] Oct 30 09:43:06.030883 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:06.030857 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7037b5f-57852"] Oct 30 09:43:06.275744 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:06.275698 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145760b0-8e56-4019-ad04-f0aedf4265be" path="/var/lib/kubelet/pods/145760b0-8e56-4019-ad04-f0aedf4265be/volumes" Oct 30 09:43:12.268187 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.268153 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29"] Oct 30 09:43:12.268986 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.268969 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d5563d9a-edff-4a2e-9dcd-001e876851f8" containerName="revoke-keys" Oct 30 09:43:12.269045 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.268988 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5563d9a-edff-4a2e-9dcd-001e876851f8" containerName="revoke-keys" Oct 30 09:43:12.269123 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.269113 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5563d9a-edff-4a2e-9dcd-001e876851f8" containerName="revoke-keys" Oct 30 09:43:12.273141 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.273108 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:12.291725 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.291694 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29"] Oct 30 09:43:12.301648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.301043 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a525bd\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a525bd\"" Oct 30 09:43:12.301648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.301043 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a525bd\"/\"kube-root-ca.crt\"" Oct 30 09:43:12.301648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.301209 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a525bd\"/\"openshift-service-ca.crt\"" Oct 30 09:43:12.303943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.303908 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a525bd\"/\"agc-cred-ce91eabdb1d5\"" Oct 30 09:43:12.304211 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.303908 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a525bd\"/\"routing-overlay\"" Oct 30 09:43:12.304211 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.303957 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a525bd\"/\"payload-processing-external-model-e2e--49b58304-dockercfg-2xsrq\"" Oct 30 09:43:12.376146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.376097 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dww9\" (UniqueName: \"kubernetes.io/projected/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-kube-api-access-6dww9\") pod \"payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29\" (UID: \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\") " pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:12.376359 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.376225 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29\" (UID: \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\") " pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:12.376359 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.376310 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29\" (UID: \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\") " pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:12.376452 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.376420 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29\" (UID: \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\") " pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:12.478721 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.478405 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dww9\" (UniqueName: \"kubernetes.io/projected/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-kube-api-access-6dww9\") pod \"payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29\" (UID: \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\") " pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:12.479557 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.479511 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29\" (UID: \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\") " pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:12.479759 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.479629 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29\" (UID: \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\") " pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:12.479759 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.479714 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29\" (UID: \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\") " pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:12.480974 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.480901 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29\" (UID: \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\") " pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:12.512612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.491631 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29\" (UID: \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\") " pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:12.523696 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.523543 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dww9\" (UniqueName: \"kubernetes.io/projected/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-kube-api-access-6dww9\") pod \"payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29\" (UID: \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\") " pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:12.537646 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.525679 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29\" (UID: \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\") " pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:12.591410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.591370 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:12.904364 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:12.904331 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29"] Oct 30 09:43:12.905353 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:43:12.905320 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1f16cc_1d04_4f6b_b246_c1bcb2342efc.slice/crio-ecbd4b19550c3911969014f1bbcb98688b53a5e3422e17b9887d73f6739c3c93 WatchSource:0}: Error finding container ecbd4b19550c3911969014f1bbcb98688b53a5e3422e17b9887d73f6739c3c93: Status 404 returned error can't find the container with id ecbd4b19550c3911969014f1bbcb98688b53a5e3422e17b9887d73f6739c3c93 Oct 30 09:43:13.555406 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:13.555368 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl"] Oct 30 09:43:13.785111 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:13.785064 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" event={"ID":"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc","Type":"ContainerStarted","Data":"dbbc951ec020bbb47c40e2491f122bebe7011d31e42cfaa0f4cc75864ca8d861"} Oct 30 09:43:13.785314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:13.785154 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:13.785314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:13.785179 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" event={"ID":"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc","Type":"ContainerStarted","Data":"ecbd4b19550c3911969014f1bbcb98688b53a5e3422e17b9887d73f6739c3c93"} Oct 30 09:43:13.830535 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:13.830400 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" podStartSLOduration=1.830380227 podStartE2EDuration="1.830380227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:43:13.825606659 +0000 UTC m=+2911.688312471" watchObservedRunningTime="2026-10-30 09:43:13.830380227 +0000 UTC m=+2911.693086048" Oct 30 09:43:15.813191 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:43:15.813144 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:43:18.556146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:18.556084 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" podUID="60ad7415-a69c-4663-a924-5aa3a63423fe" containerName="payload-processing" containerID="cri-o://0898c797bb7d003478c113f638758f32cec3b924ef3f157731b9219fc5319d11" gracePeriod=25 Oct 30 09:43:18.830536 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:18.830507 5938 generic.go:356] "Generic (PLEG): container finished" podID="60ad7415-a69c-4663-a924-5aa3a63423fe" containerID="0898c797bb7d003478c113f638758f32cec3b924ef3f157731b9219fc5319d11" exitCode=0 Oct 30 09:43:18.830737 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:18.830631 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" event={"ID":"60ad7415-a69c-4663-a924-5aa3a63423fe","Type":"ContainerDied","Data":"0898c797bb7d003478c113f638758f32cec3b924ef3f157731b9219fc5319d11"} Oct 30 09:43:18.869934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:18.869901 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:43:19.002357 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.002304 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ad7415-a69c-4663-a924-5aa3a63423fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ad7415-a69c-4663-a924-5aa3a63423fe-plugins-config-volume\") pod \"60ad7415-a69c-4663-a924-5aa3a63423fe\" (UID: \"60ad7415-a69c-4663-a924-5aa3a63423fe\") " Oct 30 09:43:19.002547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.002431 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ad7415-a69c-4663-a924-5aa3a63423fe-kube-api-access-fpcv7\" (UniqueName: \"kubernetes.io/projected/60ad7415-a69c-4663-a924-5aa3a63423fe-kube-api-access-fpcv7\") pod \"60ad7415-a69c-4663-a924-5aa3a63423fe\" (UID: \"60ad7415-a69c-4663-a924-5aa3a63423fe\") " Oct 30 09:43:19.002743 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.002719 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ad7415-a69c-4663-a924-5aa3a63423fe-provider-credentials\" (UniqueName: \"kubernetes.io/projected/60ad7415-a69c-4663-a924-5aa3a63423fe-provider-credentials\") pod \"60ad7415-a69c-4663-a924-5aa3a63423fe\" (UID: \"60ad7415-a69c-4663-a924-5aa3a63423fe\") " Oct 30 09:43:19.002882 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.002807 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ad7415-a69c-4663-a924-5aa3a63423fe-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/60ad7415-a69c-4663-a924-5aa3a63423fe-routing-overlay-volume\") pod \"60ad7415-a69c-4663-a924-5aa3a63423fe\" (UID: \"60ad7415-a69c-4663-a924-5aa3a63423fe\") " Oct 30 09:43:19.003237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.003194 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ad7415-a69c-4663-a924-5aa3a63423fe-plugins-config-volume" pod "60ad7415-a69c-4663-a924-5aa3a63423fe" (UID: "60ad7415-a69c-4663-a924-5aa3a63423fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:43:19.003854 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.003826 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ad7415-a69c-4663-a924-5aa3a63423fe-routing-overlay-volume" pod "60ad7415-a69c-4663-a924-5aa3a63423fe" (UID: "60ad7415-a69c-4663-a924-5aa3a63423fe"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:43:19.004266 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.004240 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/60ad7415-a69c-4663-a924-5aa3a63423fe-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:19.004499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.004482 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ad7415-a69c-4663-a924-5aa3a63423fe-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:19.013087 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.013038 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ad7415-a69c-4663-a924-5aa3a63423fe-provider-credentials" pod "60ad7415-a69c-4663-a924-5aa3a63423fe" (UID: "60ad7415-a69c-4663-a924-5aa3a63423fe"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:43:19.028066 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.027676 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ad7415-a69c-4663-a924-5aa3a63423fe-kube-api-access-fpcv7" pod "60ad7415-a69c-4663-a924-5aa3a63423fe" (UID: "60ad7415-a69c-4663-a924-5aa3a63423fe"). InnerVolumeSpecName "kube-api-access-fpcv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:43:19.106050 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.105941 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/60ad7415-a69c-4663-a924-5aa3a63423fe-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:19.106050 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.105982 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpcv7\" (UniqueName: \"kubernetes.io/projected/60ad7415-a69c-4663-a924-5aa3a63423fe-kube-api-access-fpcv7\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:19.166877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.166840 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29"] Oct 30 09:43:19.170363 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.170331 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:19.841761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.841722 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" Oct 30 09:43:19.841761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.841745 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl" event={"ID":"60ad7415-a69c-4663-a924-5aa3a63423fe","Type":"ContainerDied","Data":"112367a4a772f73733fbcb32a3e200be946d4c01182243dbb83613c30c312b8f"} Oct 30 09:43:19.842284 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.841817 5938 scope.go:122] "RemoveContainer" containerID="0898c797bb7d003478c113f638758f32cec3b924ef3f157731b9219fc5319d11" Oct 30 09:43:19.897833 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.897790 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl"] Oct 30 09:43:19.912177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:19.912141 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ae795b/payload-processing-external-model-e2e-extmod-a-ae795b-6cbdtprnl"] Oct 30 09:43:20.260299 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:20.260257 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ad7415-a69c-4663-a924-5aa3a63423fe" path="/var/lib/kubelet/pods/60ad7415-a69c-4663-a924-5aa3a63423fe/volumes" Oct 30 09:43:21.965086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:21.965049 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x"] Oct 30 09:43:21.965826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:21.965811 5938 cpu_manager.go:414] "containerMap: removing container" podUID="60ad7415-a69c-4663-a924-5aa3a63423fe" containerName="payload-processing" Oct 30 09:43:21.965879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:21.965828 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ad7415-a69c-4663-a924-5aa3a63423fe" containerName="payload-processing" Oct 30 09:43:21.965975 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:21.965965 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ad7415-a69c-4663-a924-5aa3a63423fe" containerName="payload-processing" Oct 30 09:43:21.971143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:21.971119 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x" Oct 30 09:43:21.989204 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:21.989172 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x"] Oct 30 09:43:21.998368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:21.998335 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:43:21.998568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:21.998464 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:43:22.147477 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:22.147428 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fefff5c-b885-468b-bada-a1de738ab257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab755ac3-zzv6x\" (UID: \"7fefff5c-b885-468b-bada-a1de738ab257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x" Oct 30 09:43:22.147477 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:22.147479 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9wbv\" (UniqueName: \"kubernetes.io/projected/7fefff5c-b885-468b-bada-a1de738ab257-kube-api-access-t9wbv\") pod \"maas-api-revoke-sub-ab755ac3-zzv6x\" (UID: \"7fefff5c-b885-468b-bada-a1de738ab257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x" Oct 30 09:43:22.147797 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:22.147706 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fefff5c-b885-468b-bada-a1de738ab257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab755ac3-zzv6x\" (UID: \"7fefff5c-b885-468b-bada-a1de738ab257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x" Oct 30 09:43:22.249418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:22.249325 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fefff5c-b885-468b-bada-a1de738ab257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab755ac3-zzv6x\" (UID: \"7fefff5c-b885-468b-bada-a1de738ab257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x" Oct 30 09:43:22.249647 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:22.249493 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fefff5c-b885-468b-bada-a1de738ab257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab755ac3-zzv6x\" (UID: \"7fefff5c-b885-468b-bada-a1de738ab257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x" Oct 30 09:43:22.249647 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:22.249520 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9wbv\" (UniqueName: \"kubernetes.io/projected/7fefff5c-b885-468b-bada-a1de738ab257-kube-api-access-t9wbv\") pod \"maas-api-revoke-sub-ab755ac3-zzv6x\" (UID: \"7fefff5c-b885-468b-bada-a1de738ab257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x" Oct 30 09:43:22.250280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:22.250258 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fefff5c-b885-468b-bada-a1de738ab257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab755ac3-zzv6x\" (UID: \"7fefff5c-b885-468b-bada-a1de738ab257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x" Oct 30 09:43:22.252280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:22.252255 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fefff5c-b885-468b-bada-a1de738ab257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab755ac3-zzv6x\" (UID: \"7fefff5c-b885-468b-bada-a1de738ab257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x" Oct 30 09:43:22.275905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:22.275871 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9wbv\" (UniqueName: \"kubernetes.io/projected/7fefff5c-b885-468b-bada-a1de738ab257-kube-api-access-t9wbv\") pod \"maas-api-revoke-sub-ab755ac3-zzv6x\" (UID: \"7fefff5c-b885-468b-bada-a1de738ab257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x" Oct 30 09:43:22.288520 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:22.288492 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x" Oct 30 09:43:22.598232 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:22.598196 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x"] Oct 30 09:43:22.600409 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:43:22.600372 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fefff5c_b885_468b_bada_a1de738ab257.slice/crio-6679e12786ee495ad00ee41810bde99fe99f2c920b19b83fd5d9d05d650d9460 WatchSource:0}: Error finding container 6679e12786ee495ad00ee41810bde99fe99f2c920b19b83fd5d9d05d650d9460: Status 404 returned error can't find the container with id 6679e12786ee495ad00ee41810bde99fe99f2c920b19b83fd5d9d05d650d9460 Oct 30 09:43:22.869860 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:22.869818 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x" event={"ID":"7fefff5c-b885-468b-bada-a1de738ab257","Type":"ContainerStarted","Data":"6679e12786ee495ad00ee41810bde99fe99f2c920b19b83fd5d9d05d650d9460"} Oct 30 09:43:23.879818 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:23.879784 5938 generic.go:356] "Generic (PLEG): container finished" podID="7fefff5c-b885-468b-bada-a1de738ab257" containerID="0d243b6e97b4b8418c50e0cd57187a56b72a3721fa5125931392e5665d2fb2cc" exitCode=0 Oct 30 09:43:23.880259 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:23.879833 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x" event={"ID":"7fefff5c-b885-468b-bada-a1de738ab257","Type":"ContainerDied","Data":"0d243b6e97b4b8418c50e0cd57187a56b72a3721fa5125931392e5665d2fb2cc"} Oct 30 09:43:24.167450 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.167340 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" podUID="0a1f16cc-1d04-4f6b-b246-c1bcb2342efc" containerName="payload-processing" containerID="cri-o://dbbc951ec020bbb47c40e2491f122bebe7011d31e42cfaa0f4cc75864ca8d861" gracePeriod=25 Oct 30 09:43:24.461511 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.461477 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:24.484956 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.484916 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-routing-overlay-volume\") pod \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\" (UID: \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\") " Oct 30 09:43:24.485161 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.485016 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-plugins-config-volume\") pod \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\" (UID: \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\") " Oct 30 09:43:24.485161 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.485131 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-kube-api-access-6dww9\" (UniqueName: \"kubernetes.io/projected/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-kube-api-access-6dww9\") pod \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\" (UID: \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\") " Oct 30 09:43:24.485267 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.485238 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-provider-credentials\") pod \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\" (UID: \"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc\") " Oct 30 09:43:24.485355 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.485329 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-routing-overlay-volume" pod "0a1f16cc-1d04-4f6b-b246-c1bcb2342efc" (UID: "0a1f16cc-1d04-4f6b-b246-c1bcb2342efc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:43:24.485467 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.485439 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-plugins-config-volume" pod "0a1f16cc-1d04-4f6b-b246-c1bcb2342efc" (UID: "0a1f16cc-1d04-4f6b-b246-c1bcb2342efc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:43:24.485906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.485882 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:24.485906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.485906 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:24.487808 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.487781 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-provider-credentials" pod "0a1f16cc-1d04-4f6b-b246-c1bcb2342efc" (UID: "0a1f16cc-1d04-4f6b-b246-c1bcb2342efc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:43:24.487892 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.487825 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-kube-api-access-6dww9" pod "0a1f16cc-1d04-4f6b-b246-c1bcb2342efc" (UID: "0a1f16cc-1d04-4f6b-b246-c1bcb2342efc"). InnerVolumeSpecName "kube-api-access-6dww9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:43:24.587934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.587890 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:24.587934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.587928 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dww9\" (UniqueName: \"kubernetes.io/projected/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc-kube-api-access-6dww9\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:24.888969 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.888931 5938 generic.go:356] "Generic (PLEG): container finished" podID="0a1f16cc-1d04-4f6b-b246-c1bcb2342efc" containerID="dbbc951ec020bbb47c40e2491f122bebe7011d31e42cfaa0f4cc75864ca8d861" exitCode=0 Oct 30 09:43:24.889446 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.888989 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" Oct 30 09:43:24.889446 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.889022 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" event={"ID":"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc","Type":"ContainerDied","Data":"dbbc951ec020bbb47c40e2491f122bebe7011d31e42cfaa0f4cc75864ca8d861"} Oct 30 09:43:24.889446 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.889089 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29" event={"ID":"0a1f16cc-1d04-4f6b-b246-c1bcb2342efc","Type":"ContainerDied","Data":"ecbd4b19550c3911969014f1bbcb98688b53a5e3422e17b9887d73f6739c3c93"} Oct 30 09:43:24.889446 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.889113 5938 scope.go:122] "RemoveContainer" containerID="dbbc951ec020bbb47c40e2491f122bebe7011d31e42cfaa0f4cc75864ca8d861" Oct 30 09:43:24.902267 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.902244 5938 scope.go:122] "RemoveContainer" containerID="dbbc951ec020bbb47c40e2491f122bebe7011d31e42cfaa0f4cc75864ca8d861" Oct 30 09:43:24.902716 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:43:24.902687 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbc951ec020bbb47c40e2491f122bebe7011d31e42cfaa0f4cc75864ca8d861\": container with ID starting with dbbc951ec020bbb47c40e2491f122bebe7011d31e42cfaa0f4cc75864ca8d861 not found: ID does not exist" containerID="dbbc951ec020bbb47c40e2491f122bebe7011d31e42cfaa0f4cc75864ca8d861" Oct 30 09:43:24.902807 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.902731 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbc951ec020bbb47c40e2491f122bebe7011d31e42cfaa0f4cc75864ca8d861"} err="failed to get container status \"dbbc951ec020bbb47c40e2491f122bebe7011d31e42cfaa0f4cc75864ca8d861\": rpc error: code = NotFound desc = could not find container \"dbbc951ec020bbb47c40e2491f122bebe7011d31e42cfaa0f4cc75864ca8d861\": container with ID starting with dbbc951ec020bbb47c40e2491f122bebe7011d31e42cfaa0f4cc75864ca8d861 not found: ID does not exist" Oct 30 09:43:24.947335 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.947297 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29"] Oct 30 09:43:24.961228 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:24.961187 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a525bd/payload-processing-external-model-e2e-extmod-a-a525bd-644dswb29"] Oct 30 09:43:25.063540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:25.063504 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x" Oct 30 09:43:25.095818 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:25.095770 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fefff5c-b885-468b-bada-a1de738ab257-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fefff5c-b885-468b-bada-a1de738ab257-maas-api-openshift-ca\") pod \"7fefff5c-b885-468b-bada-a1de738ab257\" (UID: \"7fefff5c-b885-468b-bada-a1de738ab257\") " Oct 30 09:43:25.096043 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:25.095854 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fefff5c-b885-468b-bada-a1de738ab257-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fefff5c-b885-468b-bada-a1de738ab257-maas-api-xks-ca\") pod \"7fefff5c-b885-468b-bada-a1de738ab257\" (UID: \"7fefff5c-b885-468b-bada-a1de738ab257\") " Oct 30 09:43:25.096043 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:25.095916 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fefff5c-b885-468b-bada-a1de738ab257-kube-api-access-t9wbv\" (UniqueName: \"kubernetes.io/projected/7fefff5c-b885-468b-bada-a1de738ab257-kube-api-access-t9wbv\") pod \"7fefff5c-b885-468b-bada-a1de738ab257\" (UID: \"7fefff5c-b885-468b-bada-a1de738ab257\") " Oct 30 09:43:25.096272 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:25.096246 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fefff5c-b885-468b-bada-a1de738ab257-maas-api-openshift-ca" pod "7fefff5c-b885-468b-bada-a1de738ab257" (UID: "7fefff5c-b885-468b-bada-a1de738ab257"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:43:25.096639 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:25.096619 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fefff5c-b885-468b-bada-a1de738ab257-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:25.098393 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:25.098366 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fefff5c-b885-468b-bada-a1de738ab257-maas-api-xks-ca" pod "7fefff5c-b885-468b-bada-a1de738ab257" (UID: "7fefff5c-b885-468b-bada-a1de738ab257"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:43:25.098819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:25.098784 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fefff5c-b885-468b-bada-a1de738ab257-kube-api-access-t9wbv" pod "7fefff5c-b885-468b-bada-a1de738ab257" (UID: "7fefff5c-b885-468b-bada-a1de738ab257"). InnerVolumeSpecName "kube-api-access-t9wbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:43:25.197890 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:25.197793 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fefff5c-b885-468b-bada-a1de738ab257-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:25.197890 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:25.197827 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9wbv\" (UniqueName: \"kubernetes.io/projected/7fefff5c-b885-468b-bada-a1de738ab257-kube-api-access-t9wbv\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:25.899532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:25.899497 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x" Oct 30 09:43:25.900048 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:25.899529 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x" event={"ID":"7fefff5c-b885-468b-bada-a1de738ab257","Type":"ContainerDied","Data":"6679e12786ee495ad00ee41810bde99fe99f2c920b19b83fd5d9d05d650d9460"} Oct 30 09:43:25.900048 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:25.899619 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6679e12786ee495ad00ee41810bde99fe99f2c920b19b83fd5d9d05d650d9460" Oct 30 09:43:26.259229 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:26.259143 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1f16cc-1d04-4f6b-b246-c1bcb2342efc" path="/var/lib/kubelet/pods/0a1f16cc-1d04-4f6b-b246-c1bcb2342efc/volumes" Oct 30 09:43:43.721558 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:43.721516 5938 scope.go:122] "RemoveContainer" containerID="bc3a6d96874b488ae9ad907f7a93ac7266ac12bc00f9270f84ff87a8cb5a5131" Oct 30 09:43:43.733871 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:43.733843 5938 scope.go:122] "RemoveContainer" containerID="5f65cfa20445e1800f97b151e42be24f5ca5ad066a8f98c5608169b3f8270c48" Oct 30 09:43:45.864882 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:43:45.864843 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:43:46.862603 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.862546 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn"] Oct 30 09:43:46.863343 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.863325 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0a1f16cc-1d04-4f6b-b246-c1bcb2342efc" containerName="payload-processing" Oct 30 09:43:46.863400 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.863344 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1f16cc-1d04-4f6b-b246-c1bcb2342efc" containerName="payload-processing" Oct 30 09:43:46.863400 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.863354 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7fefff5c-b885-468b-bada-a1de738ab257" containerName="revoke-keys" Oct 30 09:43:46.863400 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.863359 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fefff5c-b885-468b-bada-a1de738ab257" containerName="revoke-keys" Oct 30 09:43:46.863498 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.863458 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1f16cc-1d04-4f6b-b246-c1bcb2342efc" containerName="payload-processing" Oct 30 09:43:46.863498 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.863466 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fefff5c-b885-468b-bada-a1de738ab257" containerName="revoke-keys" Oct 30 09:43:46.868489 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.868458 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:46.884872 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.884817 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"routing-overlay\"" Oct 30 09:43:46.898946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.898641 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"kube-root-ca.crt\"" Oct 30 09:43:46.898946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.898756 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"openshift-service-ca.crt\"" Oct 30 09:43:46.898946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.898833 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"agc-cred-444d26bf7c74\"" Oct 30 09:43:46.898946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.898881 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d60c68\"" Oct 30 09:43:46.898946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.898912 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"payload-processing-external-model-e2e--2110ea34-dockercfg-wnxqk\"" Oct 30 09:43:46.903201 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.903163 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn"] Oct 30 09:43:46.972618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.972564 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e3451bb-3183-4ed1-babd-d9ef450cd85d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn\" (UID: \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:46.972825 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.972656 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e3451bb-3183-4ed1-babd-d9ef450cd85d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn\" (UID: \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:46.972825 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.972717 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e3451bb-3183-4ed1-babd-d9ef450cd85d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn\" (UID: \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:46.972825 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:46.972749 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmrql\" (UniqueName: \"kubernetes.io/projected/4e3451bb-3183-4ed1-babd-d9ef450cd85d-kube-api-access-wmrql\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn\" (UID: \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:47.074348 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:47.074302 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e3451bb-3183-4ed1-babd-d9ef450cd85d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn\" (UID: \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:47.074582 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:47.074518 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e3451bb-3183-4ed1-babd-d9ef450cd85d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn\" (UID: \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:47.074739 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:47.074716 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e3451bb-3183-4ed1-babd-d9ef450cd85d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn\" (UID: \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:47.074838 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:47.074820 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmrql\" (UniqueName: \"kubernetes.io/projected/4e3451bb-3183-4ed1-babd-d9ef450cd85d-kube-api-access-wmrql\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn\" (UID: \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:47.075540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:47.075496 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e3451bb-3183-4ed1-babd-d9ef450cd85d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn\" (UID: \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:47.076064 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:47.076038 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e3451bb-3183-4ed1-babd-d9ef450cd85d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn\" (UID: \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:47.078874 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:47.078840 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e3451bb-3183-4ed1-babd-d9ef450cd85d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn\" (UID: \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:47.162681 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:47.162611 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmrql\" (UniqueName: \"kubernetes.io/projected/4e3451bb-3183-4ed1-babd-d9ef450cd85d-kube-api-access-wmrql\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn\" (UID: \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:47.184433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:47.184393 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:47.506366 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:47.506336 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn"] Oct 30 09:43:47.509232 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:43:47.509200 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3451bb_3183_4ed1_babd_d9ef450cd85d.slice/crio-aaa22da0ba4317b9961bc6e4faf426f0573ec0bf34af54d082c0ed79566711b1 WatchSource:0}: Error finding container aaa22da0ba4317b9961bc6e4faf426f0573ec0bf34af54d082c0ed79566711b1: Status 404 returned error can't find the container with id aaa22da0ba4317b9961bc6e4faf426f0573ec0bf34af54d082c0ed79566711b1 Oct 30 09:43:48.183921 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:48.183878 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" event={"ID":"4e3451bb-3183-4ed1-babd-d9ef450cd85d","Type":"ContainerStarted","Data":"1dfb033e17c09186608b2737a934fe3a871cc48209a829ca784174a9deb54c5c"} Oct 30 09:43:48.184375 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:48.183939 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" event={"ID":"4e3451bb-3183-4ed1-babd-d9ef450cd85d","Type":"ContainerStarted","Data":"aaa22da0ba4317b9961bc6e4faf426f0573ec0bf34af54d082c0ed79566711b1"} Oct 30 09:43:48.184375 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:48.183965 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:48.230517 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:48.230444 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" podStartSLOduration=2.230425543 podStartE2EDuration="2.230425543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:43:48.226508315 +0000 UTC m=+2946.089214136" watchObservedRunningTime="2026-10-30 09:43:48.230425543 +0000 UTC m=+2946.093131379" Oct 30 09:43:49.536293 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.536239 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-74f944f77-jfq98"] Oct 30 09:43:49.541699 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.541666 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:43:49.560205 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.560164 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-74f944f77-jfq98"] Oct 30 09:43:49.568019 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.567988 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 09:43:49.568318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.567988 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 09:43:49.568318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.567988 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 09:43:49.568318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.568108 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 09:43:49.571672 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.571638 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ff81f0cb94c6\"" Oct 30 09:43:49.571672 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.571669 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-dbd224b134a2\"" Oct 30 09:43:49.571977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.571711 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pwntl\"" Oct 30 09:43:49.615128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.615079 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6d0da42-776d-4a4a-980d-c2494e623bc9-routing-overlay-volume\") pod \"payload-processing-external-model-74f944f77-jfq98\" (UID: \"a6d0da42-776d-4a4a-980d-c2494e623bc9\") " pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:43:49.615354 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.615158 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6d0da42-776d-4a4a-980d-c2494e623bc9-provider-credentials\") pod \"payload-processing-external-model-74f944f77-jfq98\" (UID: \"a6d0da42-776d-4a4a-980d-c2494e623bc9\") " pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:43:49.615410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.615339 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2nwb\" (UniqueName: \"kubernetes.io/projected/a6d0da42-776d-4a4a-980d-c2494e623bc9-kube-api-access-z2nwb\") pod \"payload-processing-external-model-74f944f77-jfq98\" (UID: \"a6d0da42-776d-4a4a-980d-c2494e623bc9\") " pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:43:49.615460 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.615438 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d0da42-776d-4a4a-980d-c2494e623bc9-plugins-config-volume\") pod \"payload-processing-external-model-74f944f77-jfq98\" (UID: \"a6d0da42-776d-4a4a-980d-c2494e623bc9\") " pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:43:49.717387 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.717342 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6d0da42-776d-4a4a-980d-c2494e623bc9-routing-overlay-volume\") pod \"payload-processing-external-model-74f944f77-jfq98\" (UID: \"a6d0da42-776d-4a4a-980d-c2494e623bc9\") " pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:43:49.717638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.717404 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6d0da42-776d-4a4a-980d-c2494e623bc9-provider-credentials\") pod \"payload-processing-external-model-74f944f77-jfq98\" (UID: \"a6d0da42-776d-4a4a-980d-c2494e623bc9\") " pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:43:49.717638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.717446 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2nwb\" (UniqueName: \"kubernetes.io/projected/a6d0da42-776d-4a4a-980d-c2494e623bc9-kube-api-access-z2nwb\") pod \"payload-processing-external-model-74f944f77-jfq98\" (UID: \"a6d0da42-776d-4a4a-980d-c2494e623bc9\") " pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:43:49.717638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.717480 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d0da42-776d-4a4a-980d-c2494e623bc9-plugins-config-volume\") pod \"payload-processing-external-model-74f944f77-jfq98\" (UID: \"a6d0da42-776d-4a4a-980d-c2494e623bc9\") " pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:43:49.718192 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.718164 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6d0da42-776d-4a4a-980d-c2494e623bc9-routing-overlay-volume\") pod \"payload-processing-external-model-74f944f77-jfq98\" (UID: \"a6d0da42-776d-4a4a-980d-c2494e623bc9\") " pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:43:49.718474 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.718200 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d0da42-776d-4a4a-980d-c2494e623bc9-plugins-config-volume\") pod \"payload-processing-external-model-74f944f77-jfq98\" (UID: \"a6d0da42-776d-4a4a-980d-c2494e623bc9\") " pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:43:49.720380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.720353 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6d0da42-776d-4a4a-980d-c2494e623bc9-provider-credentials\") pod \"payload-processing-external-model-74f944f77-jfq98\" (UID: \"a6d0da42-776d-4a4a-980d-c2494e623bc9\") " pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:43:49.739951 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.739921 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2nwb\" (UniqueName: \"kubernetes.io/projected/a6d0da42-776d-4a4a-980d-c2494e623bc9-kube-api-access-z2nwb\") pod \"payload-processing-external-model-74f944f77-jfq98\" (UID: \"a6d0da42-776d-4a4a-980d-c2494e623bc9\") " pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:43:49.863774 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:49.863742 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:43:50.175140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:50.175102 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-74f944f77-jfq98"] Oct 30 09:43:50.176270 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:43:50.176241 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d0da42_776d_4a4a_980d_c2494e623bc9.slice/crio-022209142ec8cc11ad30a976899ad0ecd0744ef723db8e7773d3bca2579f83f9 WatchSource:0}: Error finding container 022209142ec8cc11ad30a976899ad0ecd0744ef723db8e7773d3bca2579f83f9: Status 404 returned error can't find the container with id 022209142ec8cc11ad30a976899ad0ecd0744ef723db8e7773d3bca2579f83f9 Oct 30 09:43:50.202521 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:50.202485 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" event={"ID":"a6d0da42-776d-4a4a-980d-c2494e623bc9","Type":"ContainerStarted","Data":"022209142ec8cc11ad30a976899ad0ecd0744ef723db8e7773d3bca2579f83f9"} Oct 30 09:43:51.214080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:51.214031 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" event={"ID":"a6d0da42-776d-4a4a-980d-c2494e623bc9","Type":"ContainerStarted","Data":"54e2fa9d516e4d2c8bbf0149a68268a445d89aa2adaff3428348829a742bba00"} Oct 30 09:43:51.214548 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:51.214113 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:43:51.261956 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:51.261884 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" podStartSLOduration=2.261867644 podStartE2EDuration="2.261867644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:43:51.257637694 +0000 UTC m=+2949.120343515" watchObservedRunningTime="2026-10-30 09:43:51.261867644 +0000 UTC m=+2949.124573557" Oct 30 09:43:54.031522 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:54.031481 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t"] Oct 30 09:43:54.044463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:54.044426 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d598e56-h9r9t"] Oct 30 09:43:54.178288 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:54.178249 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn"] Oct 30 09:43:54.181711 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:54.181687 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:54.262372 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:54.262329 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3e34684-56f1-4eae-ba7d-0b7abbdcff8e" path="/var/lib/kubelet/pods/a3e34684-56f1-4eae-ba7d-0b7abbdcff8e/volumes" Oct 30 09:43:56.667437 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:56.667153 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p"] Oct 30 09:43:56.674515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:56.674484 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" Oct 30 09:43:56.690098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:56.690067 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p"] Oct 30 09:43:56.699246 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:56.699214 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d66d22-dockercfg-bb8bj\"" Oct 30 09:43:56.737210 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:56.737172 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d66d22\"" Oct 30 09:43:56.841912 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:56.841867 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n255w\" (UniqueName: \"kubernetes.io/projected/245627dd-ae75-45aa-9f14-af4797b0d87c-kube-api-access-n255w\") pod \"payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p\" (UID: \"245627dd-ae75-45aa-9f14-af4797b0d87c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" Oct 30 09:43:56.842389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:56.842359 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/245627dd-ae75-45aa-9f14-af4797b0d87c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p\" (UID: \"245627dd-ae75-45aa-9f14-af4797b0d87c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" Oct 30 09:43:56.944019 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:56.943910 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/245627dd-ae75-45aa-9f14-af4797b0d87c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p\" (UID: \"245627dd-ae75-45aa-9f14-af4797b0d87c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" Oct 30 09:43:56.944223 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:56.944153 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n255w\" (UniqueName: \"kubernetes.io/projected/245627dd-ae75-45aa-9f14-af4797b0d87c-kube-api-access-n255w\") pod \"payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p\" (UID: \"245627dd-ae75-45aa-9f14-af4797b0d87c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" Oct 30 09:43:56.944723 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:56.944705 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/245627dd-ae75-45aa-9f14-af4797b0d87c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p\" (UID: \"245627dd-ae75-45aa-9f14-af4797b0d87c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" Oct 30 09:43:56.965621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:56.965546 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n255w\" (UniqueName: \"kubernetes.io/projected/245627dd-ae75-45aa-9f14-af4797b0d87c-kube-api-access-n255w\") pod \"payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p\" (UID: \"245627dd-ae75-45aa-9f14-af4797b0d87c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" Oct 30 09:43:56.992549 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:56.992516 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" Oct 30 09:43:57.303661 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:57.303626 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p"] Oct 30 09:43:57.305005 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:43:57.304971 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245627dd_ae75_45aa_9f14_af4797b0d87c.slice/crio-1f6963992d8751f69e174f9227f477444691480ab2ed49dc00e0b984dd6bc5a3 WatchSource:0}: Error finding container 1f6963992d8751f69e174f9227f477444691480ab2ed49dc00e0b984dd6bc5a3: Status 404 returned error can't find the container with id 1f6963992d8751f69e174f9227f477444691480ab2ed49dc00e0b984dd6bc5a3 Oct 30 09:43:57.906405 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:57.906367 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx"] Oct 30 09:43:57.906942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:57.906604 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" podUID="a464a5b4-c19b-4e64-8b9b-7b8459471501" containerName="istio-proxy" containerID="cri-o://07223fadf24fdcf584ad275072c2ce4e9017cfecb07d6368c0092f416a11e930" gracePeriod=30 Oct 30 09:43:58.281751 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:58.281653 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" event={"ID":"245627dd-ae75-45aa-9f14-af4797b0d87c","Type":"ContainerStarted","Data":"ae49e4592e70694f4a4154ab6fee1cc87c7182945a3b38811b7f41dc5aeb5fc1"} Oct 30 09:43:58.281751 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:58.281737 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" event={"ID":"245627dd-ae75-45aa-9f14-af4797b0d87c","Type":"ContainerStarted","Data":"1f6963992d8751f69e174f9227f477444691480ab2ed49dc00e0b984dd6bc5a3"} Oct 30 09:43:58.281977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:58.281913 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" Oct 30 09:43:58.328279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:58.328220 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" podStartSLOduration=2.328205385 podStartE2EDuration="2.328205385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:43:58.325462866 +0000 UTC m=+2956.188168712" watchObservedRunningTime="2026-10-30 09:43:58.328205385 +0000 UTC m=+2956.190911205" Oct 30 09:43:59.179294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:59.179212 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" podUID="4e3451bb-3183-4ed1-babd-d9ef450cd85d" containerName="payload-processing" containerID="cri-o://1dfb033e17c09186608b2737a934fe3a871cc48209a829ca784174a9deb54c5c" gracePeriod=25 Oct 30 09:43:59.482573 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:59.482544 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:43:59.583514 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:59.583478 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e3451bb-3183-4ed1-babd-d9ef450cd85d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e3451bb-3183-4ed1-babd-d9ef450cd85d-plugins-config-volume\") pod \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\" (UID: \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\") " Oct 30 09:43:59.583726 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:59.583559 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e3451bb-3183-4ed1-babd-d9ef450cd85d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e3451bb-3183-4ed1-babd-d9ef450cd85d-routing-overlay-volume\") pod \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\" (UID: \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\") " Oct 30 09:43:59.583726 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:59.583661 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3451bb-3183-4ed1-babd-d9ef450cd85d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e3451bb-3183-4ed1-babd-d9ef450cd85d-provider-credentials\") pod \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\" (UID: \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\") " Oct 30 09:43:59.583878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:59.583787 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3451bb-3183-4ed1-babd-d9ef450cd85d-kube-api-access-wmrql\" (UniqueName: \"kubernetes.io/projected/4e3451bb-3183-4ed1-babd-d9ef450cd85d-kube-api-access-wmrql\") pod \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\" (UID: \"4e3451bb-3183-4ed1-babd-d9ef450cd85d\") " Oct 30 09:43:59.583941 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:59.583917 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3451bb-3183-4ed1-babd-d9ef450cd85d-routing-overlay-volume" pod "4e3451bb-3183-4ed1-babd-d9ef450cd85d" (UID: "4e3451bb-3183-4ed1-babd-d9ef450cd85d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:43:59.583994 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:59.583957 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3451bb-3183-4ed1-babd-d9ef450cd85d-plugins-config-volume" pod "4e3451bb-3183-4ed1-babd-d9ef450cd85d" (UID: "4e3451bb-3183-4ed1-babd-d9ef450cd85d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:43:59.584521 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:59.584499 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e3451bb-3183-4ed1-babd-d9ef450cd85d-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:59.584580 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:59.584525 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e3451bb-3183-4ed1-babd-d9ef450cd85d-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:59.586200 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:59.586177 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3451bb-3183-4ed1-babd-d9ef450cd85d-provider-credentials" pod "4e3451bb-3183-4ed1-babd-d9ef450cd85d" (UID: "4e3451bb-3183-4ed1-babd-d9ef450cd85d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:43:59.586270 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:59.586233 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3451bb-3183-4ed1-babd-d9ef450cd85d-kube-api-access-wmrql" pod "4e3451bb-3183-4ed1-babd-d9ef450cd85d" (UID: "4e3451bb-3183-4ed1-babd-d9ef450cd85d"). InnerVolumeSpecName "kube-api-access-wmrql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:43:59.686323 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:59.686272 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmrql\" (UniqueName: \"kubernetes.io/projected/4e3451bb-3183-4ed1-babd-d9ef450cd85d-kube-api-access-wmrql\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:43:59.686323 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:43:59.686319 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e3451bb-3183-4ed1-babd-d9ef450cd85d-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:00.299532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:00.299500 5938 generic.go:356] "Generic (PLEG): container finished" podID="4e3451bb-3183-4ed1-babd-d9ef450cd85d" containerID="1dfb033e17c09186608b2737a934fe3a871cc48209a829ca784174a9deb54c5c" exitCode=0 Oct 30 09:44:00.300056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:00.299556 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" Oct 30 09:44:00.300056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:00.299581 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" event={"ID":"4e3451bb-3183-4ed1-babd-d9ef450cd85d","Type":"ContainerDied","Data":"1dfb033e17c09186608b2737a934fe3a871cc48209a829ca784174a9deb54c5c"} Oct 30 09:44:00.300056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:00.299674 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn" event={"ID":"4e3451bb-3183-4ed1-babd-d9ef450cd85d","Type":"ContainerDied","Data":"aaa22da0ba4317b9961bc6e4faf426f0573ec0bf34af54d082c0ed79566711b1"} Oct 30 09:44:00.300056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:00.299705 5938 scope.go:122] "RemoveContainer" containerID="1dfb033e17c09186608b2737a934fe3a871cc48209a829ca784174a9deb54c5c" Oct 30 09:44:00.310560 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:00.310537 5938 scope.go:122] "RemoveContainer" containerID="1dfb033e17c09186608b2737a934fe3a871cc48209a829ca784174a9deb54c5c" Oct 30 09:44:00.310910 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:44:00.310885 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfb033e17c09186608b2737a934fe3a871cc48209a829ca784174a9deb54c5c\": container with ID starting with 1dfb033e17c09186608b2737a934fe3a871cc48209a829ca784174a9deb54c5c not found: ID does not exist" containerID="1dfb033e17c09186608b2737a934fe3a871cc48209a829ca784174a9deb54c5c" Oct 30 09:44:00.310988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:00.310924 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfb033e17c09186608b2737a934fe3a871cc48209a829ca784174a9deb54c5c"} err="failed to get container status \"1dfb033e17c09186608b2737a934fe3a871cc48209a829ca784174a9deb54c5c\": rpc error: code = NotFound desc = could not find container \"1dfb033e17c09186608b2737a934fe3a871cc48209a829ca784174a9deb54c5c\": container with ID starting with 1dfb033e17c09186608b2737a934fe3a871cc48209a829ca784174a9deb54c5c not found: ID does not exist" Oct 30 09:44:00.357561 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:00.357526 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn"] Oct 30 09:44:00.371218 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:00.371189 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-5b56m65tn"] Oct 30 09:44:02.227335 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:02.227308 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:44:02.262183 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:02.262144 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e3451bb-3183-4ed1-babd-d9ef450cd85d" path="/var/lib/kubelet/pods/4e3451bb-3183-4ed1-babd-d9ef450cd85d/volumes" Oct 30 09:44:03.206953 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.206927 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:44:03.241068 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.241034 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-workload-socket\") pod \"a464a5b4-c19b-4e64-8b9b-7b8459471501\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " Oct 30 09:44:03.241631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.241095 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-token\" (UniqueName: \"kubernetes.io/projected/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-token\") pod \"a464a5b4-c19b-4e64-8b9b-7b8459471501\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " Oct 30 09:44:03.241631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.241145 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-credential-socket\") pod \"a464a5b4-c19b-4e64-8b9b-7b8459471501\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " Oct 30 09:44:03.241631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.241169 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-workload-certs\") pod \"a464a5b4-c19b-4e64-8b9b-7b8459471501\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " Oct 30 09:44:03.241631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.241228 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a464a5b4-c19b-4e64-8b9b-7b8459471501-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a464a5b4-c19b-4e64-8b9b-7b8459471501-istiod-ca-cert\") pod \"a464a5b4-c19b-4e64-8b9b-7b8459471501\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " Oct 30 09:44:03.241631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.241291 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-envoy\") pod \"a464a5b4-c19b-4e64-8b9b-7b8459471501\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " Oct 30 09:44:03.241631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.241376 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-data\") pod \"a464a5b4-c19b-4e64-8b9b-7b8459471501\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " Oct 30 09:44:03.241631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.241418 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-workload-socket" pod "a464a5b4-c19b-4e64-8b9b-7b8459471501" (UID: "a464a5b4-c19b-4e64-8b9b-7b8459471501"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:44:03.241631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.241516 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-credential-socket" pod "a464a5b4-c19b-4e64-8b9b-7b8459471501" (UID: "a464a5b4-c19b-4e64-8b9b-7b8459471501"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:44:03.241631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.241549 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a464a5b4-c19b-4e64-8b9b-7b8459471501-kube-api-access-k5dtb\" (UniqueName: \"kubernetes.io/projected/a464a5b4-c19b-4e64-8b9b-7b8459471501-kube-api-access-k5dtb\") pod \"a464a5b4-c19b-4e64-8b9b-7b8459471501\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " Oct 30 09:44:03.242131 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.241690 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-podinfo\") pod \"a464a5b4-c19b-4e64-8b9b-7b8459471501\" (UID: \"a464a5b4-c19b-4e64-8b9b-7b8459471501\") " Oct 30 09:44:03.242131 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.241703 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-data" pod "a464a5b4-c19b-4e64-8b9b-7b8459471501" (UID: "a464a5b4-c19b-4e64-8b9b-7b8459471501"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:44:03.242131 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.241741 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a464a5b4-c19b-4e64-8b9b-7b8459471501-istiod-ca-cert" pod "a464a5b4-c19b-4e64-8b9b-7b8459471501" (UID: "a464a5b4-c19b-4e64-8b9b-7b8459471501"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:44:03.242131 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.241910 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-workload-certs" pod "a464a5b4-c19b-4e64-8b9b-7b8459471501" (UID: "a464a5b4-c19b-4e64-8b9b-7b8459471501"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:44:03.242623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.242573 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:03.242623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.242619 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:03.242751 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.242635 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:03.242751 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.242650 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:03.242751 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.242663 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a464a5b4-c19b-4e64-8b9b-7b8459471501-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:03.244069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.244040 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a464a5b4-c19b-4e64-8b9b-7b8459471501-kube-api-access-k5dtb" pod "a464a5b4-c19b-4e64-8b9b-7b8459471501" (UID: "a464a5b4-c19b-4e64-8b9b-7b8459471501"). InnerVolumeSpecName "kube-api-access-k5dtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:03.244429 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.244409 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-token" pod "a464a5b4-c19b-4e64-8b9b-7b8459471501" (UID: "a464a5b4-c19b-4e64-8b9b-7b8459471501"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:03.244499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.244462 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-podinfo" pod "a464a5b4-c19b-4e64-8b9b-7b8459471501" (UID: "a464a5b4-c19b-4e64-8b9b-7b8459471501"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:44:03.244539 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.244488 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-envoy" pod "a464a5b4-c19b-4e64-8b9b-7b8459471501" (UID: "a464a5b4-c19b-4e64-8b9b-7b8459471501"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:44:03.330160 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.330120 5938 generic.go:356] "Generic (PLEG): container finished" podID="a464a5b4-c19b-4e64-8b9b-7b8459471501" containerID="07223fadf24fdcf584ad275072c2ce4e9017cfecb07d6368c0092f416a11e930" exitCode=0 Oct 30 09:44:03.330433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.330187 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" Oct 30 09:44:03.330433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.330209 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" event={"ID":"a464a5b4-c19b-4e64-8b9b-7b8459471501","Type":"ContainerDied","Data":"07223fadf24fdcf584ad275072c2ce4e9017cfecb07d6368c0092f416a11e930"} Oct 30 09:44:03.330433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.330300 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" event={"ID":"a464a5b4-c19b-4e64-8b9b-7b8459471501","Type":"ContainerDied","Data":"953229ad21718d0cf8ca822730cc3a440a9c2b4e11c1975e02af9e3d32d75679"} Oct 30 09:44:03.330433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.330330 5938 scope.go:122] "RemoveContainer" containerID="07223fadf24fdcf584ad275072c2ce4e9017cfecb07d6368c0092f416a11e930" Oct 30 09:44:03.342125 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.342102 5938 scope.go:122] "RemoveContainer" containerID="07223fadf24fdcf584ad275072c2ce4e9017cfecb07d6368c0092f416a11e930" Oct 30 09:44:03.342448 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:44:03.342429 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07223fadf24fdcf584ad275072c2ce4e9017cfecb07d6368c0092f416a11e930\": container with ID starting with 07223fadf24fdcf584ad275072c2ce4e9017cfecb07d6368c0092f416a11e930 not found: ID does not exist" containerID="07223fadf24fdcf584ad275072c2ce4e9017cfecb07d6368c0092f416a11e930" Oct 30 09:44:03.342500 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.342459 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07223fadf24fdcf584ad275072c2ce4e9017cfecb07d6368c0092f416a11e930"} err="failed to get container status \"07223fadf24fdcf584ad275072c2ce4e9017cfecb07d6368c0092f416a11e930\": rpc error: code = NotFound desc = could not find container \"07223fadf24fdcf584ad275072c2ce4e9017cfecb07d6368c0092f416a11e930\": container with ID starting with 07223fadf24fdcf584ad275072c2ce4e9017cfecb07d6368c0092f416a11e930 not found: ID does not exist" Oct 30 09:44:03.344343 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.344324 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:03.344395 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.344347 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:03.344395 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.344358 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a464a5b4-c19b-4e64-8b9b-7b8459471501-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:03.344395 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.344367 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5dtb\" (UniqueName: \"kubernetes.io/projected/a464a5b4-c19b-4e64-8b9b-7b8459471501-kube-api-access-k5dtb\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:03.395058 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.395001 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx"] Oct 30 09:44:03.400198 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:03.400169 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx"] Oct 30 09:44:04.125894 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:04.125850 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:44:04.126103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:04.125944 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-vsskx" podUID="a464a5b4-c19b-4e64-8b9b-7b8459471501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:44:04.260517 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:04.260471 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a464a5b4-c19b-4e64-8b9b-7b8459471501" path="/var/lib/kubelet/pods/a464a5b4-c19b-4e64-8b9b-7b8459471501/volumes" Oct 30 09:44:05.030497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:05.030457 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct"] Oct 30 09:44:05.043486 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:05.043455 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c3c0b4-78kct"] Oct 30 09:44:06.261672 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:06.261639 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d0eda9a-f78d-4624-8514-0b4f2a1531e8" path="/var/lib/kubelet/pods/3d0eda9a-f78d-4624-8514-0b4f2a1531e8/volumes" Oct 30 09:44:09.292968 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:09.292934 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" Oct 30 09:44:10.405905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.405870 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc"] Oct 30 09:44:10.406719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.406689 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a464a5b4-c19b-4e64-8b9b-7b8459471501" containerName="istio-proxy" Oct 30 09:44:10.406719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.406713 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a464a5b4-c19b-4e64-8b9b-7b8459471501" containerName="istio-proxy" Oct 30 09:44:10.406912 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.406746 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4e3451bb-3183-4ed1-babd-d9ef450cd85d" containerName="payload-processing" Oct 30 09:44:10.406912 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.406751 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3451bb-3183-4ed1-babd-d9ef450cd85d" containerName="payload-processing" Oct 30 09:44:10.406912 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.406899 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a464a5b4-c19b-4e64-8b9b-7b8459471501" containerName="istio-proxy" Oct 30 09:44:10.406912 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.406911 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3451bb-3183-4ed1-babd-d9ef450cd85d" containerName="payload-processing" Oct 30 09:44:10.412181 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.412161 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.428814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.428767 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-j7g5p\"" Oct 30 09:44:10.446945 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.446907 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc"] Oct 30 09:44:10.544787 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.544744 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.544984 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.544813 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.544984 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.544939 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.545121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.545041 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.545121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.545086 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.545189 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.545150 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.545189 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.545185 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.545322 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.545302 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxqmq\" (UniqueName: \"kubernetes.io/projected/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-kube-api-access-dxqmq\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.545388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.545359 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.647061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.647026 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.647280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.647080 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.647280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.647137 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.647280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.647198 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.647280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.647228 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.647532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.647280 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.647532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.647314 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.647532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.647375 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxqmq\" (UniqueName: \"kubernetes.io/projected/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-kube-api-access-dxqmq\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.647532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.647414 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.647784 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.647691 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.647784 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.647702 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.647907 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.647879 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.647970 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.647888 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.648230 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.648210 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.649739 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.649723 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.649977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.649958 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.669173 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.669097 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.669319 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.669296 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxqmq\" (UniqueName: \"kubernetes.io/projected/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-kube-api-access-dxqmq\") pod \"maas-default-gateway-openshift-default-77f556d647-hxpzc\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:10.727916 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:10.727877 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:11.064423 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:11.064390 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc"] Oct 30 09:44:11.066066 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:44:11.066031 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b49bb6_bb6c_4c24_9c9a_1fd74645757f.slice/crio-321cd4d584bb85f573c442c5f95ff95b7b64c2795d9fd206a3e354cb819815e8 WatchSource:0}: Error finding container 321cd4d584bb85f573c442c5f95ff95b7b64c2795d9fd206a3e354cb819815e8: Status 404 returned error can't find the container with id 321cd4d584bb85f573c442c5f95ff95b7b64c2795d9fd206a3e354cb819815e8 Oct 30 09:44:11.069079 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:11.069043 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:44:11.069196 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:11.069137 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:44:11.069257 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:11.069197 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:44:11.395683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:11.395643 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" event={"ID":"02b49bb6-bb6c-4c24-9c9a-1fd74645757f","Type":"ContainerStarted","Data":"b684b1173841f3ea13bd6c000334fa8e479f1ecb2e6aa0bad135d7a5e32fddd8"} Oct 30 09:44:11.395899 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:11.395724 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" event={"ID":"02b49bb6-bb6c-4c24-9c9a-1fd74645757f","Type":"ContainerStarted","Data":"321cd4d584bb85f573c442c5f95ff95b7b64c2795d9fd206a3e354cb819815e8"} Oct 30 09:44:11.444969 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:11.444905 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" podStartSLOduration=1.44488895 podStartE2EDuration="1.44488895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:44:11.441789661 +0000 UTC m=+2969.304495517" watchObservedRunningTime="2026-10-30 09:44:11.44488895 +0000 UTC m=+2969.307594771" Oct 30 09:44:11.729079 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:11.728976 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:12.745490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:12.745452 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hxpzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 09:44:12.745951 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:12.745509 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" podUID="02b49bb6-bb6c-4c24-9c9a-1fd74645757f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 09:44:13.744263 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:13.744215 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hxpzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 09:44:13.744508 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:13.744288 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" podUID="02b49bb6-bb6c-4c24-9c9a-1fd74645757f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 09:44:14.738616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:14.738043 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hxpzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 09:44:14.738616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:14.738113 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" podUID="02b49bb6-bb6c-4c24-9c9a-1fd74645757f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 09:44:15.743618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:15.739069 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hxpzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 09:44:15.743618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:15.739139 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" podUID="02b49bb6-bb6c-4c24-9c9a-1fd74645757f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 09:44:15.844935 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:44:15.844896 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:44:16.740617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:16.740040 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hxpzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 09:44:16.740617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:16.740129 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" podUID="02b49bb6-bb6c-4c24-9c9a-1fd74645757f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 09:44:17.728712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:17.728669 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hxpzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 09:44:17.729144 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:17.728738 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" podUID="02b49bb6-bb6c-4c24-9c9a-1fd74645757f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 09:44:18.729024 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:18.728990 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hxpzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 09:44:18.729514 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:18.729049 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" podUID="02b49bb6-bb6c-4c24-9c9a-1fd74645757f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 09:44:19.906165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:19.906128 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:19.907982 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:19.906874 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:19.908151 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:19.908113 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 09:44:24.401445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:24.401356 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs"] Oct 30 09:44:24.406788 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:24.406767 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" Oct 30 09:44:24.427278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:24.427238 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs"] Oct 30 09:44:24.436046 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:24.436021 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:44:24.436217 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:24.436198 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:44:24.482812 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:24.482779 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e82e336-f8ad-43f2-a417-32432fa7192c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de59b49-8jkxs\" (UID: \"6e82e336-f8ad-43f2-a417-32432fa7192c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" Oct 30 09:44:24.483092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:24.483072 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e82e336-f8ad-43f2-a417-32432fa7192c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de59b49-8jkxs\" (UID: \"6e82e336-f8ad-43f2-a417-32432fa7192c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" Oct 30 09:44:24.483139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:24.483116 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7nf\" (UniqueName: \"kubernetes.io/projected/6e82e336-f8ad-43f2-a417-32432fa7192c-kube-api-access-qn7nf\") pod \"maas-api-revoke-sub-8de59b49-8jkxs\" (UID: \"6e82e336-f8ad-43f2-a417-32432fa7192c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" Oct 30 09:44:24.584045 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:24.584011 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e82e336-f8ad-43f2-a417-32432fa7192c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de59b49-8jkxs\" (UID: \"6e82e336-f8ad-43f2-a417-32432fa7192c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" Oct 30 09:44:24.584247 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:24.584070 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7nf\" (UniqueName: \"kubernetes.io/projected/6e82e336-f8ad-43f2-a417-32432fa7192c-kube-api-access-qn7nf\") pod \"maas-api-revoke-sub-8de59b49-8jkxs\" (UID: \"6e82e336-f8ad-43f2-a417-32432fa7192c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" Oct 30 09:44:24.584247 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:24.584112 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e82e336-f8ad-43f2-a417-32432fa7192c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de59b49-8jkxs\" (UID: \"6e82e336-f8ad-43f2-a417-32432fa7192c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" Oct 30 09:44:24.584952 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:24.584922 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e82e336-f8ad-43f2-a417-32432fa7192c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de59b49-8jkxs\" (UID: \"6e82e336-f8ad-43f2-a417-32432fa7192c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" Oct 30 09:44:24.586647 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:24.586626 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e82e336-f8ad-43f2-a417-32432fa7192c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de59b49-8jkxs\" (UID: \"6e82e336-f8ad-43f2-a417-32432fa7192c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" Oct 30 09:44:24.606865 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:24.606835 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7nf\" (UniqueName: \"kubernetes.io/projected/6e82e336-f8ad-43f2-a417-32432fa7192c-kube-api-access-qn7nf\") pod \"maas-api-revoke-sub-8de59b49-8jkxs\" (UID: \"6e82e336-f8ad-43f2-a417-32432fa7192c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" Oct 30 09:44:24.722907 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:24.722821 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" Oct 30 09:44:25.029511 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.029470 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs"] Oct 30 09:44:25.029798 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:44:25.029777 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e82e336_f8ad_43f2_a417_32432fa7192c.slice/crio-965437a3547034461dabfa0336681adde016353f1a55638ebdfa7375f4432c48 WatchSource:0}: Error finding container 965437a3547034461dabfa0336681adde016353f1a55638ebdfa7375f4432c48: Status 404 returned error can't find the container with id 965437a3547034461dabfa0336681adde016353f1a55638ebdfa7375f4432c48 Oct 30 09:44:25.469739 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.469698 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr"] Oct 30 09:44:25.474487 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.474465 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.506635 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.498297 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr"] Oct 30 09:44:25.540902 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.540859 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" event={"ID":"6e82e336-f8ad-43f2-a417-32432fa7192c","Type":"ContainerStarted","Data":"1eb89d0d532a063e5b06caf7c160aeb58e34e493d34b7c5dd5c80f4fadc32a04"} Oct 30 09:44:25.541088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.540933 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" event={"ID":"6e82e336-f8ad-43f2-a417-32432fa7192c","Type":"ContainerStarted","Data":"965437a3547034461dabfa0336681adde016353f1a55638ebdfa7375f4432c48"} Oct 30 09:44:25.595656 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.595522 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" podStartSLOduration=1.595501708 podStartE2EDuration="1.595501708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:44:25.591501873 +0000 UTC m=+2983.454207718" watchObservedRunningTime="2026-10-30 09:44:25.595501708 +0000 UTC m=+2983.458207529" Oct 30 09:44:25.600347 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.600313 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38877f2-7419-459b-a30c-8a55b146cb48-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.600526 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.600507 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38877f2-7419-459b-a30c-8a55b146cb48-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.600605 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.600555 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjgnz\" (UniqueName: \"kubernetes.io/projected/f38877f2-7419-459b-a30c-8a55b146cb48-kube-api-access-hjgnz\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.600835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.600811 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38877f2-7419-459b-a30c-8a55b146cb48-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.600949 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.600855 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38877f2-7419-459b-a30c-8a55b146cb48-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.600949 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.600932 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38877f2-7419-459b-a30c-8a55b146cb48-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.601063 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.600981 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38877f2-7419-459b-a30c-8a55b146cb48-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.601152 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.601134 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f38877f2-7419-459b-a30c-8a55b146cb48-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.601190 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.601165 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38877f2-7419-459b-a30c-8a55b146cb48-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.702818 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.702779 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f38877f2-7419-459b-a30c-8a55b146cb48-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.702818 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.702821 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38877f2-7419-459b-a30c-8a55b146cb48-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.703059 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.702843 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38877f2-7419-459b-a30c-8a55b146cb48-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.703059 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.702914 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38877f2-7419-459b-a30c-8a55b146cb48-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.703059 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.702944 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjgnz\" (UniqueName: \"kubernetes.io/projected/f38877f2-7419-459b-a30c-8a55b146cb48-kube-api-access-hjgnz\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.703059 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.703017 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38877f2-7419-459b-a30c-8a55b146cb48-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.703284 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.703162 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38877f2-7419-459b-a30c-8a55b146cb48-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.703284 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.703220 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38877f2-7419-459b-a30c-8a55b146cb48-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.703284 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.703272 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38877f2-7419-459b-a30c-8a55b146cb48-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.703486 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.703449 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38877f2-7419-459b-a30c-8a55b146cb48-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.703608 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.703564 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38877f2-7419-459b-a30c-8a55b146cb48-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.703683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.703635 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38877f2-7419-459b-a30c-8a55b146cb48-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.703735 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.703676 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38877f2-7419-459b-a30c-8a55b146cb48-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.703833 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.703812 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38877f2-7419-459b-a30c-8a55b146cb48-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.705781 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.705758 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38877f2-7419-459b-a30c-8a55b146cb48-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.705986 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.705968 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38877f2-7419-459b-a30c-8a55b146cb48-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.727790 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.727754 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f38877f2-7419-459b-a30c-8a55b146cb48-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.729914 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.729893 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjgnz\" (UniqueName: \"kubernetes.io/projected/f38877f2-7419-459b-a30c-8a55b146cb48-kube-api-access-hjgnz\") pod \"maas-default-gateway-openshift-default-77f556d647-vnxfr\" (UID: \"f38877f2-7419-459b-a30c-8a55b146cb48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:25.791785 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:25.791747 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:26.114708 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:26.114678 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr"] Oct 30 09:44:26.115728 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:44:26.115696 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38877f2_7419_459b_a30c_8a55b146cb48.slice/crio-cf2884f582a56c048ca3fff5c0789ca4166153294daeb67dbf1f914a0093db10 WatchSource:0}: Error finding container cf2884f582a56c048ca3fff5c0789ca4166153294daeb67dbf1f914a0093db10: Status 404 returned error can't find the container with id cf2884f582a56c048ca3fff5c0789ca4166153294daeb67dbf1f914a0093db10 Oct 30 09:44:26.119256 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:26.119210 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:44:26.119409 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:26.119313 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:44:26.119409 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:26.119359 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:44:26.551615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:26.551564 5938 generic.go:356] "Generic (PLEG): container finished" podID="6e82e336-f8ad-43f2-a417-32432fa7192c" containerID="1eb89d0d532a063e5b06caf7c160aeb58e34e493d34b7c5dd5c80f4fadc32a04" exitCode=0 Oct 30 09:44:26.552083 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:26.551714 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" event={"ID":"6e82e336-f8ad-43f2-a417-32432fa7192c","Type":"ContainerDied","Data":"1eb89d0d532a063e5b06caf7c160aeb58e34e493d34b7c5dd5c80f4fadc32a04"} Oct 30 09:44:26.553645 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:26.553614 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" event={"ID":"f38877f2-7419-459b-a30c-8a55b146cb48","Type":"ContainerStarted","Data":"abb268dbc7de23bf6a5fcd3eae530dc2e1ba5d552fde711df3b53c510b144cd6"} Oct 30 09:44:26.553800 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:26.553664 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" event={"ID":"f38877f2-7419-459b-a30c-8a55b146cb48","Type":"ContainerStarted","Data":"cf2884f582a56c048ca3fff5c0789ca4166153294daeb67dbf1f914a0093db10"} Oct 30 09:44:26.651972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:26.651909 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" podStartSLOduration=1.6518871910000001 podStartE2EDuration="1.651887191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:44:26.648917453 +0000 UTC m=+2984.511623278" watchObservedRunningTime="2026-10-30 09:44:26.651887191 +0000 UTC m=+2984.514593013" Oct 30 09:44:26.791880 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:26.791845 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:27.771032 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:27.771001 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" Oct 30 09:44:27.794366 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:27.794332 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vnxfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 09:44:27.794521 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:27.794384 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" podUID="f38877f2-7419-459b-a30c-8a55b146cb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 09:44:27.852218 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:27.852117 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e82e336-f8ad-43f2-a417-32432fa7192c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e82e336-f8ad-43f2-a417-32432fa7192c-maas-api-xks-ca\") pod \"6e82e336-f8ad-43f2-a417-32432fa7192c\" (UID: \"6e82e336-f8ad-43f2-a417-32432fa7192c\") " Oct 30 09:44:27.852375 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:27.852257 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e82e336-f8ad-43f2-a417-32432fa7192c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e82e336-f8ad-43f2-a417-32432fa7192c-maas-api-openshift-ca\") pod \"6e82e336-f8ad-43f2-a417-32432fa7192c\" (UID: \"6e82e336-f8ad-43f2-a417-32432fa7192c\") " Oct 30 09:44:27.852375 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:27.852348 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e82e336-f8ad-43f2-a417-32432fa7192c-kube-api-access-qn7nf\" (UniqueName: \"kubernetes.io/projected/6e82e336-f8ad-43f2-a417-32432fa7192c-kube-api-access-qn7nf\") pod \"6e82e336-f8ad-43f2-a417-32432fa7192c\" (UID: \"6e82e336-f8ad-43f2-a417-32432fa7192c\") " Oct 30 09:44:27.853074 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:27.853049 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e82e336-f8ad-43f2-a417-32432fa7192c-maas-api-openshift-ca" pod "6e82e336-f8ad-43f2-a417-32432fa7192c" (UID: "6e82e336-f8ad-43f2-a417-32432fa7192c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:44:27.853259 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:27.853233 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e82e336-f8ad-43f2-a417-32432fa7192c-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:27.855468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:27.855439 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e82e336-f8ad-43f2-a417-32432fa7192c-kube-api-access-qn7nf" pod "6e82e336-f8ad-43f2-a417-32432fa7192c" (UID: "6e82e336-f8ad-43f2-a417-32432fa7192c"). InnerVolumeSpecName "kube-api-access-qn7nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:27.866359 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:27.866330 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e82e336-f8ad-43f2-a417-32432fa7192c-maas-api-xks-ca" pod "6e82e336-f8ad-43f2-a417-32432fa7192c" (UID: "6e82e336-f8ad-43f2-a417-32432fa7192c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:44:27.955094 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:27.955058 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e82e336-f8ad-43f2-a417-32432fa7192c-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:27.955094 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:27.955091 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn7nf\" (UniqueName: \"kubernetes.io/projected/6e82e336-f8ad-43f2-a417-32432fa7192c-kube-api-access-qn7nf\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:28.640613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:28.639332 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" Oct 30 09:44:28.640613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:28.640079 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs" event={"ID":"6e82e336-f8ad-43f2-a417-32432fa7192c","Type":"ContainerDied","Data":"965437a3547034461dabfa0336681adde016353f1a55638ebdfa7375f4432c48"} Oct 30 09:44:28.640613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:28.640127 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="965437a3547034461dabfa0336681adde016353f1a55638ebdfa7375f4432c48" Oct 30 09:44:28.792969 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:28.792932 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vnxfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 09:44:28.793349 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:28.792982 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" podUID="f38877f2-7419-459b-a30c-8a55b146cb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 09:44:29.792438 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:29.792397 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vnxfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 09:44:29.792669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:29.792463 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" podUID="f38877f2-7419-459b-a30c-8a55b146cb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 09:44:30.792651 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:30.792607 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vnxfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 09:44:30.793108 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:30.792680 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" podUID="f38877f2-7419-459b-a30c-8a55b146cb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 09:44:31.254072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.254031 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv"] Oct 30 09:44:31.254908 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.254883 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6e82e336-f8ad-43f2-a417-32432fa7192c" containerName="revoke-keys" Oct 30 09:44:31.254908 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.254907 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e82e336-f8ad-43f2-a417-32432fa7192c" containerName="revoke-keys" Oct 30 09:44:31.255122 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.255066 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e82e336-f8ad-43f2-a417-32432fa7192c" containerName="revoke-keys" Oct 30 09:44:31.259301 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.259283 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:44:31.280372 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.280347 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"agc-cred-896ed7620299\"" Oct 30 09:44:31.284893 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.284868 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv"] Oct 30 09:44:31.350875 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.350835 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bea1a86-26a1-4395-84dd-9c94c9e6413b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv\" (UID: \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:44:31.351200 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.351176 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bea1a86-26a1-4395-84dd-9c94c9e6413b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv\" (UID: \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:44:31.351358 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.351343 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzwxw\" (UniqueName: \"kubernetes.io/projected/1bea1a86-26a1-4395-84dd-9c94c9e6413b-kube-api-access-nzwxw\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv\" (UID: \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:44:31.351540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.351530 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bea1a86-26a1-4395-84dd-9c94c9e6413b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv\" (UID: \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:44:31.380426 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.380400 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"agc-cred-fa5ab9b30048\"" Oct 30 09:44:31.453089 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.453058 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bea1a86-26a1-4395-84dd-9c94c9e6413b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv\" (UID: \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:44:31.453298 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.453235 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bea1a86-26a1-4395-84dd-9c94c9e6413b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv\" (UID: \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:44:31.453361 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.453321 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bea1a86-26a1-4395-84dd-9c94c9e6413b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv\" (UID: \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:44:31.453422 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.453371 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzwxw\" (UniqueName: \"kubernetes.io/projected/1bea1a86-26a1-4395-84dd-9c94c9e6413b-kube-api-access-nzwxw\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv\" (UID: \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:44:31.453979 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.453953 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bea1a86-26a1-4395-84dd-9c94c9e6413b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv\" (UID: \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:44:31.454170 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.454010 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bea1a86-26a1-4395-84dd-9c94c9e6413b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv\" (UID: \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:44:31.456307 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.456280 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bea1a86-26a1-4395-84dd-9c94c9e6413b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv\" (UID: \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:44:31.488733 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.488705 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzwxw\" (UniqueName: \"kubernetes.io/projected/1bea1a86-26a1-4395-84dd-9c94c9e6413b-kube-api-access-nzwxw\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv\" (UID: \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:44:31.575440 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.575356 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:44:31.792177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.792141 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vnxfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 09:44:31.792365 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.792192 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" podUID="f38877f2-7419-459b-a30c-8a55b146cb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 09:44:31.907964 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:31.907936 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv"] Oct 30 09:44:31.909043 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:44:31.909004 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bea1a86_26a1_4395_84dd_9c94c9e6413b.slice/crio-c6db529e34228e484925c9282d19de7f3170a7e492490a1fd31f0dc368867262 WatchSource:0}: Error finding container c6db529e34228e484925c9282d19de7f3170a7e492490a1fd31f0dc368867262: Status 404 returned error can't find the container with id c6db529e34228e484925c9282d19de7f3170a7e492490a1fd31f0dc368867262 Oct 30 09:44:32.693479 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:32.693427 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" event={"ID":"1bea1a86-26a1-4395-84dd-9c94c9e6413b","Type":"ContainerStarted","Data":"52a4d5c99e5fde5e8d22aac33bdba6a289c7bec846f219dd7fac58c3a5797e12"} Oct 30 09:44:32.693728 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:32.693533 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:44:32.693728 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:32.693560 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" event={"ID":"1bea1a86-26a1-4395-84dd-9c94c9e6413b","Type":"ContainerStarted","Data":"c6db529e34228e484925c9282d19de7f3170a7e492490a1fd31f0dc368867262"} Oct 30 09:44:32.736175 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:32.736122 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" podStartSLOduration=1.736107415 podStartE2EDuration="1.736107415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:44:32.735966904 +0000 UTC m=+2990.598672730" watchObservedRunningTime="2026-10-30 09:44:32.736107415 +0000 UTC m=+2990.598813280" Oct 30 09:44:32.792458 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:32.792423 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vnxfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 09:44:32.792691 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:32.792488 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" podUID="f38877f2-7419-459b-a30c-8a55b146cb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 09:44:33.792194 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:33.792158 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vnxfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 09:44:33.792750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:33.792220 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" podUID="f38877f2-7419-459b-a30c-8a55b146cb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 09:44:35.038565 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:35.038532 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:35.039053 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:35.039031 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:35.039870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:35.039852 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vnxfr" Oct 30 09:44:36.108780 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.108750 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf"] Oct 30 09:44:36.113486 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.113465 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:36.131518 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.131495 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf"] Oct 30 09:44:36.141613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.141219 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"openshift-service-ca.crt\"" Oct 30 09:44:36.141613 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.141303 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"kube-root-ca.crt\"" Oct 30 09:44:36.143638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.143617 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"agc-cred-a64cac9b1414\"" Oct 30 09:44:36.143795 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.143637 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"payload-processing-external-model-e2e--dd1a53db-dockercfg-lznsm\"" Oct 30 09:44:36.143936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.143653 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3d9b36\"" Oct 30 09:44:36.143936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.143668 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"agc-cred-b1b80af3b434\"" Oct 30 09:44:36.143936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.143701 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"routing-overlay\"" Oct 30 09:44:36.228960 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.228928 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8455d270-3802-4b0a-b8e7-1399edbba05d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf\" (UID: \"8455d270-3802-4b0a-b8e7-1399edbba05d\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:36.229164 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.229143 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8455d270-3802-4b0a-b8e7-1399edbba05d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf\" (UID: \"8455d270-3802-4b0a-b8e7-1399edbba05d\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:36.229211 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.229186 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8455d270-3802-4b0a-b8e7-1399edbba05d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf\" (UID: \"8455d270-3802-4b0a-b8e7-1399edbba05d\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:36.229306 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.229292 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc7wj\" (UniqueName: \"kubernetes.io/projected/8455d270-3802-4b0a-b8e7-1399edbba05d-kube-api-access-rc7wj\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf\" (UID: \"8455d270-3802-4b0a-b8e7-1399edbba05d\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:36.306158 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.306125 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd"] Oct 30 09:44:36.310818 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.310796 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" Oct 30 09:44:36.327212 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.327185 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:44:36.327360 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.327307 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:44:36.330408 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.330367 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd"] Oct 30 09:44:36.331208 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.331180 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8455d270-3802-4b0a-b8e7-1399edbba05d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf\" (UID: \"8455d270-3802-4b0a-b8e7-1399edbba05d\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:36.331425 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.331406 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8455d270-3802-4b0a-b8e7-1399edbba05d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf\" (UID: \"8455d270-3802-4b0a-b8e7-1399edbba05d\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:36.331503 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.331453 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8455d270-3802-4b0a-b8e7-1399edbba05d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf\" (UID: \"8455d270-3802-4b0a-b8e7-1399edbba05d\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:36.331580 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.331566 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7wj\" (UniqueName: \"kubernetes.io/projected/8455d270-3802-4b0a-b8e7-1399edbba05d-kube-api-access-rc7wj\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf\" (UID: \"8455d270-3802-4b0a-b8e7-1399edbba05d\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:36.333402 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.333376 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8455d270-3802-4b0a-b8e7-1399edbba05d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf\" (UID: \"8455d270-3802-4b0a-b8e7-1399edbba05d\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:36.334305 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.334272 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8455d270-3802-4b0a-b8e7-1399edbba05d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf\" (UID: \"8455d270-3802-4b0a-b8e7-1399edbba05d\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:36.336435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.336412 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8455d270-3802-4b0a-b8e7-1399edbba05d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf\" (UID: \"8455d270-3802-4b0a-b8e7-1399edbba05d\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:36.356336 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.356310 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc7wj\" (UniqueName: \"kubernetes.io/projected/8455d270-3802-4b0a-b8e7-1399edbba05d-kube-api-access-rc7wj\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf\" (UID: \"8455d270-3802-4b0a-b8e7-1399edbba05d\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:36.438367 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.438071 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:36.449504 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.448018 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69cq2\" (UniqueName: \"kubernetes.io/projected/5415e392-4927-49a4-a271-ddd0487cb071-kube-api-access-69cq2\") pod \"maas-api-revoke-sub-81554c57-sdzdd\" (UID: \"5415e392-4927-49a4-a271-ddd0487cb071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" Oct 30 09:44:36.500178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.500151 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5415e392-4927-49a4-a271-ddd0487cb071-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81554c57-sdzdd\" (UID: \"5415e392-4927-49a4-a271-ddd0487cb071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" Oct 30 09:44:36.500394 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.500363 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5415e392-4927-49a4-a271-ddd0487cb071-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81554c57-sdzdd\" (UID: \"5415e392-4927-49a4-a271-ddd0487cb071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" Oct 30 09:44:36.619653 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.619614 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69cq2\" (UniqueName: \"kubernetes.io/projected/5415e392-4927-49a4-a271-ddd0487cb071-kube-api-access-69cq2\") pod \"maas-api-revoke-sub-81554c57-sdzdd\" (UID: \"5415e392-4927-49a4-a271-ddd0487cb071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" Oct 30 09:44:36.619936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.619675 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5415e392-4927-49a4-a271-ddd0487cb071-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81554c57-sdzdd\" (UID: \"5415e392-4927-49a4-a271-ddd0487cb071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" Oct 30 09:44:36.619936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.619718 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5415e392-4927-49a4-a271-ddd0487cb071-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81554c57-sdzdd\" (UID: \"5415e392-4927-49a4-a271-ddd0487cb071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" Oct 30 09:44:36.621076 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.620993 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5415e392-4927-49a4-a271-ddd0487cb071-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81554c57-sdzdd\" (UID: \"5415e392-4927-49a4-a271-ddd0487cb071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" Oct 30 09:44:36.623532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.623492 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5415e392-4927-49a4-a271-ddd0487cb071-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81554c57-sdzdd\" (UID: \"5415e392-4927-49a4-a271-ddd0487cb071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" Oct 30 09:44:36.647410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.647377 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69cq2\" (UniqueName: \"kubernetes.io/projected/5415e392-4927-49a4-a271-ddd0487cb071-kube-api-access-69cq2\") pod \"maas-api-revoke-sub-81554c57-sdzdd\" (UID: \"5415e392-4927-49a4-a271-ddd0487cb071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" Oct 30 09:44:36.836882 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:44:36.836853 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8455d270_3802_4b0a_b8e7_1399edbba05d.slice/crio-bbc891665c921da28598a9ad2b89f535917b365114b6ac0e458b61c26be24963 WatchSource:0}: Error finding container bbc891665c921da28598a9ad2b89f535917b365114b6ac0e458b61c26be24963: Status 404 returned error can't find the container with id bbc891665c921da28598a9ad2b89f535917b365114b6ac0e458b61c26be24963 Oct 30 09:44:36.837514 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.837493 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf"] Oct 30 09:44:36.924973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:36.924949 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" Oct 30 09:44:37.229180 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:37.229156 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd"] Oct 30 09:44:37.230324 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:44:37.230298 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5415e392_4927_49a4_a271_ddd0487cb071.slice/crio-dc121e8dbcd6c0b5b5f98fae19d215099d0ed4ce8809527e4e212c5e321242c8 WatchSource:0}: Error finding container dc121e8dbcd6c0b5b5f98fae19d215099d0ed4ce8809527e4e212c5e321242c8: Status 404 returned error can't find the container with id dc121e8dbcd6c0b5b5f98fae19d215099d0ed4ce8809527e4e212c5e321242c8 Oct 30 09:44:37.738688 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:37.738647 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" event={"ID":"5415e392-4927-49a4-a271-ddd0487cb071","Type":"ContainerStarted","Data":"fb76de80a843f5a268a328b67e5d508c78bf213947b8b39a2936208db892955e"} Oct 30 09:44:37.738857 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:37.738716 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" event={"ID":"5415e392-4927-49a4-a271-ddd0487cb071","Type":"ContainerStarted","Data":"dc121e8dbcd6c0b5b5f98fae19d215099d0ed4ce8809527e4e212c5e321242c8"} Oct 30 09:44:37.740407 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:37.740380 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" event={"ID":"8455d270-3802-4b0a-b8e7-1399edbba05d","Type":"ContainerStarted","Data":"47859c48adba0c3e0c665c375c2a23ad74723f4b39ebcfe365a16e4bcb743c4b"} Oct 30 09:44:37.740541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:37.740440 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:37.740541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:37.740467 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" event={"ID":"8455d270-3802-4b0a-b8e7-1399edbba05d","Type":"ContainerStarted","Data":"bbc891665c921da28598a9ad2b89f535917b365114b6ac0e458b61c26be24963"} Oct 30 09:44:37.809532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:37.809473 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" podStartSLOduration=1.80945773 podStartE2EDuration="1.80945773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:44:37.808434492 +0000 UTC m=+2995.671140313" watchObservedRunningTime="2026-10-30 09:44:37.80945773 +0000 UTC m=+2995.672163551" Oct 30 09:44:37.880092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:37.879959 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" podStartSLOduration=1.879897809 podStartE2EDuration="1.879897809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:44:37.877336449 +0000 UTC m=+2995.740042270" watchObservedRunningTime="2026-10-30 09:44:37.879897809 +0000 UTC m=+2995.742603663" Oct 30 09:44:38.749775 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:38.749737 5938 generic.go:356] "Generic (PLEG): container finished" podID="5415e392-4927-49a4-a271-ddd0487cb071" containerID="fb76de80a843f5a268a328b67e5d508c78bf213947b8b39a2936208db892955e" exitCode=0 Oct 30 09:44:38.750179 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:38.749819 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" event={"ID":"5415e392-4927-49a4-a271-ddd0487cb071","Type":"ContainerDied","Data":"fb76de80a843f5a268a328b67e5d508c78bf213947b8b39a2936208db892955e"} Oct 30 09:44:39.916382 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:39.916355 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" Oct 30 09:44:39.965828 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:39.965786 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5415e392-4927-49a4-a271-ddd0487cb071-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5415e392-4927-49a4-a271-ddd0487cb071-maas-api-openshift-ca\") pod \"5415e392-4927-49a4-a271-ddd0487cb071\" (UID: \"5415e392-4927-49a4-a271-ddd0487cb071\") " Oct 30 09:44:39.966010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:39.965991 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5415e392-4927-49a4-a271-ddd0487cb071-kube-api-access-69cq2\" (UniqueName: \"kubernetes.io/projected/5415e392-4927-49a4-a271-ddd0487cb071-kube-api-access-69cq2\") pod \"5415e392-4927-49a4-a271-ddd0487cb071\" (UID: \"5415e392-4927-49a4-a271-ddd0487cb071\") " Oct 30 09:44:39.966105 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:39.966085 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5415e392-4927-49a4-a271-ddd0487cb071-maas-api-openshift-ca" pod "5415e392-4927-49a4-a271-ddd0487cb071" (UID: "5415e392-4927-49a4-a271-ddd0487cb071"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:44:39.966198 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:39.966183 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5415e392-4927-49a4-a271-ddd0487cb071-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5415e392-4927-49a4-a271-ddd0487cb071-maas-api-xks-ca\") pod \"5415e392-4927-49a4-a271-ddd0487cb071\" (UID: \"5415e392-4927-49a4-a271-ddd0487cb071\") " Oct 30 09:44:39.966657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:39.966638 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5415e392-4927-49a4-a271-ddd0487cb071-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:39.968380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:39.968358 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5415e392-4927-49a4-a271-ddd0487cb071-maas-api-xks-ca" pod "5415e392-4927-49a4-a271-ddd0487cb071" (UID: "5415e392-4927-49a4-a271-ddd0487cb071"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:44:39.968514 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:39.968382 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5415e392-4927-49a4-a271-ddd0487cb071-kube-api-access-69cq2" pod "5415e392-4927-49a4-a271-ddd0487cb071" (UID: "5415e392-4927-49a4-a271-ddd0487cb071"). InnerVolumeSpecName "kube-api-access-69cq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:40.067748 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:40.067663 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69cq2\" (UniqueName: \"kubernetes.io/projected/5415e392-4927-49a4-a271-ddd0487cb071-kube-api-access-69cq2\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:40.067748 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:40.067700 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5415e392-4927-49a4-a271-ddd0487cb071-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:40.766354 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:40.766322 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" Oct 30 09:44:40.766562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:40.766361 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd" event={"ID":"5415e392-4927-49a4-a271-ddd0487cb071","Type":"ContainerDied","Data":"dc121e8dbcd6c0b5b5f98fae19d215099d0ed4ce8809527e4e212c5e321242c8"} Oct 30 09:44:40.766562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:40.766407 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc121e8dbcd6c0b5b5f98fae19d215099d0ed4ce8809527e4e212c5e321242c8" Oct 30 09:44:43.708676 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:43.708649 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:44:43.789080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:43.789042 5938 scope.go:122] "RemoveContainer" containerID="8699d1254cde90b303c134d5e86dd61783a6442912fdfb49475641bbfd26febf" Oct 30 09:44:43.798576 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:43.798544 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c"] Oct 30 09:44:43.803676 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:43.803648 5938 scope.go:122] "RemoveContainer" containerID="3f4506ec3143315f0165dc821e21ee44f6490cc60e6fe9c9bd0d8b91d2425361" Oct 30 09:44:43.814228 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:43.814205 5938 scope.go:122] "RemoveContainer" containerID="a5ae2fab8567cadbd7b7715b0ba5b69e2cfcb9150a46ecad5f0bc2c64b43a1bd" Oct 30 09:44:43.830199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:43.830175 5938 scope.go:122] "RemoveContainer" containerID="48b3a696e85d462d9fac1306f49e404664873cb748e3692bbf9d51119c8cf43c" Oct 30 09:44:45.813473 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:44:45.813429 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:44:48.027128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.027095 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf"] Oct 30 09:44:48.030064 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.030043 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:48.294926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.294848 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl"] Oct 30 09:44:48.295557 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.295543 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5415e392-4927-49a4-a271-ddd0487cb071" containerName="revoke-keys" Oct 30 09:44:48.295624 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.295559 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5415e392-4927-49a4-a271-ddd0487cb071" containerName="revoke-keys" Oct 30 09:44:48.295726 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.295716 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5415e392-4927-49a4-a271-ddd0487cb071" containerName="revoke-keys" Oct 30 09:44:48.301204 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.301185 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" Oct 30 09:44:48.317431 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.317402 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:44:48.317667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.317646 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:44:48.320223 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.320202 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl"] Oct 30 09:44:48.396051 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.396018 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08e92f2-4e98-4e05-936f-e11730ed863e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-675f9f2b-xdlfl\" (UID: \"f08e92f2-4e98-4e05-936f-e11730ed863e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" Oct 30 09:44:48.396234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.396074 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf5bd\" (UniqueName: \"kubernetes.io/projected/f08e92f2-4e98-4e05-936f-e11730ed863e-kube-api-access-jf5bd\") pod \"maas-api-revoke-sub-675f9f2b-xdlfl\" (UID: \"f08e92f2-4e98-4e05-936f-e11730ed863e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" Oct 30 09:44:48.396234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.396203 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08e92f2-4e98-4e05-936f-e11730ed863e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-675f9f2b-xdlfl\" (UID: \"f08e92f2-4e98-4e05-936f-e11730ed863e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" Oct 30 09:44:48.497314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.497279 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08e92f2-4e98-4e05-936f-e11730ed863e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-675f9f2b-xdlfl\" (UID: \"f08e92f2-4e98-4e05-936f-e11730ed863e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" Oct 30 09:44:48.497461 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.497333 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf5bd\" (UniqueName: \"kubernetes.io/projected/f08e92f2-4e98-4e05-936f-e11730ed863e-kube-api-access-jf5bd\") pod \"maas-api-revoke-sub-675f9f2b-xdlfl\" (UID: \"f08e92f2-4e98-4e05-936f-e11730ed863e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" Oct 30 09:44:48.497461 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.497372 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08e92f2-4e98-4e05-936f-e11730ed863e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-675f9f2b-xdlfl\" (UID: \"f08e92f2-4e98-4e05-936f-e11730ed863e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" Oct 30 09:44:48.498115 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.498091 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08e92f2-4e98-4e05-936f-e11730ed863e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-675f9f2b-xdlfl\" (UID: \"f08e92f2-4e98-4e05-936f-e11730ed863e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" Oct 30 09:44:48.500003 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.499978 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08e92f2-4e98-4e05-936f-e11730ed863e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-675f9f2b-xdlfl\" (UID: \"f08e92f2-4e98-4e05-936f-e11730ed863e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" Oct 30 09:44:48.519179 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.519153 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf5bd\" (UniqueName: \"kubernetes.io/projected/f08e92f2-4e98-4e05-936f-e11730ed863e-kube-api-access-jf5bd\") pod \"maas-api-revoke-sub-675f9f2b-xdlfl\" (UID: \"f08e92f2-4e98-4e05-936f-e11730ed863e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" Oct 30 09:44:48.617068 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.617034 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" Oct 30 09:44:48.799391 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.799349 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" podUID="bd6cf721-374c-4a4e-b31f-17a351c4adf3" containerName="payload-processing" containerID="cri-o://742a8e6935792d5e4a8aa63b2648511ba5d8eeb81a615a1378db574fb96cd4e7" gracePeriod=25 Oct 30 09:44:48.924179 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:44:48.924024 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08e92f2_4e98_4e05_936f_e11730ed863e.slice/crio-56efe7f5a21c8346f06bf2e82fb6bcc52b004d4466d2a7fae51b2f8c205f29d9 WatchSource:0}: Error finding container 56efe7f5a21c8346f06bf2e82fb6bcc52b004d4466d2a7fae51b2f8c205f29d9: Status 404 returned error can't find the container with id 56efe7f5a21c8346f06bf2e82fb6bcc52b004d4466d2a7fae51b2f8c205f29d9 Oct 30 09:44:48.926059 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:48.925817 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl"] Oct 30 09:44:49.130799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.130772 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:44:49.205503 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.205428 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd6cf721-374c-4a4e-b31f-17a351c4adf3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6cf721-374c-4a4e-b31f-17a351c4adf3-plugins-config-volume\") pod \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\" (UID: \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\") " Oct 30 09:44:49.205503 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.205473 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd6cf721-374c-4a4e-b31f-17a351c4adf3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd6cf721-374c-4a4e-b31f-17a351c4adf3-routing-overlay-volume\") pod \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\" (UID: \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\") " Oct 30 09:44:49.205742 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.205520 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6cf721-374c-4a4e-b31f-17a351c4adf3-kube-api-access-zv6xc\" (UniqueName: \"kubernetes.io/projected/bd6cf721-374c-4a4e-b31f-17a351c4adf3-kube-api-access-zv6xc\") pod \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\" (UID: \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\") " Oct 30 09:44:49.205846 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.205815 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6cf721-374c-4a4e-b31f-17a351c4adf3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd6cf721-374c-4a4e-b31f-17a351c4adf3-provider-credentials\") pod \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\" (UID: \"bd6cf721-374c-4a4e-b31f-17a351c4adf3\") " Oct 30 09:44:49.205959 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.205940 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6cf721-374c-4a4e-b31f-17a351c4adf3-plugins-config-volume" pod "bd6cf721-374c-4a4e-b31f-17a351c4adf3" (UID: "bd6cf721-374c-4a4e-b31f-17a351c4adf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:44:49.206025 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.205971 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6cf721-374c-4a4e-b31f-17a351c4adf3-routing-overlay-volume" pod "bd6cf721-374c-4a4e-b31f-17a351c4adf3" (UID: "bd6cf721-374c-4a4e-b31f-17a351c4adf3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:44:49.206439 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.206416 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6cf721-374c-4a4e-b31f-17a351c4adf3-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:49.206439 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.206437 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd6cf721-374c-4a4e-b31f-17a351c4adf3-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:49.208116 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.208089 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6cf721-374c-4a4e-b31f-17a351c4adf3-kube-api-access-zv6xc" pod "bd6cf721-374c-4a4e-b31f-17a351c4adf3" (UID: "bd6cf721-374c-4a4e-b31f-17a351c4adf3"). InnerVolumeSpecName "kube-api-access-zv6xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:49.208762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.208741 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6cf721-374c-4a4e-b31f-17a351c4adf3-provider-credentials" pod "bd6cf721-374c-4a4e-b31f-17a351c4adf3" (UID: "bd6cf721-374c-4a4e-b31f-17a351c4adf3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:49.307556 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.307523 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd6cf721-374c-4a4e-b31f-17a351c4adf3-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:49.307556 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.307549 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv6xc\" (UniqueName: \"kubernetes.io/projected/bd6cf721-374c-4a4e-b31f-17a351c4adf3-kube-api-access-zv6xc\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:49.838287 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.838199 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" event={"ID":"f08e92f2-4e98-4e05-936f-e11730ed863e","Type":"ContainerStarted","Data":"bde2c52a8bae69afd7dff15efe5fbca351274e76370a665397fb6c01431c8054"} Oct 30 09:44:49.838287 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.838266 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" event={"ID":"f08e92f2-4e98-4e05-936f-e11730ed863e","Type":"ContainerStarted","Data":"56efe7f5a21c8346f06bf2e82fb6bcc52b004d4466d2a7fae51b2f8c205f29d9"} Oct 30 09:44:49.839701 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.839670 5938 generic.go:356] "Generic (PLEG): container finished" podID="bd6cf721-374c-4a4e-b31f-17a351c4adf3" containerID="742a8e6935792d5e4a8aa63b2648511ba5d8eeb81a615a1378db574fb96cd4e7" exitCode=0 Oct 30 09:44:49.839837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.839722 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" Oct 30 09:44:49.839837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.839757 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" event={"ID":"bd6cf721-374c-4a4e-b31f-17a351c4adf3","Type":"ContainerDied","Data":"742a8e6935792d5e4a8aa63b2648511ba5d8eeb81a615a1378db574fb96cd4e7"} Oct 30 09:44:49.839837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.839810 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c" event={"ID":"bd6cf721-374c-4a4e-b31f-17a351c4adf3","Type":"ContainerDied","Data":"cc479c356568b14ba48f1891fcdcc0245a01610636a1863a37afda0ab8feedec"} Oct 30 09:44:49.839837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.839832 5938 scope.go:122] "RemoveContainer" containerID="742a8e6935792d5e4a8aa63b2648511ba5d8eeb81a615a1378db574fb96cd4e7" Oct 30 09:44:49.850750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.850731 5938 scope.go:122] "RemoveContainer" containerID="742a8e6935792d5e4a8aa63b2648511ba5d8eeb81a615a1378db574fb96cd4e7" Oct 30 09:44:49.851038 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:44:49.851019 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"742a8e6935792d5e4a8aa63b2648511ba5d8eeb81a615a1378db574fb96cd4e7\": container with ID starting with 742a8e6935792d5e4a8aa63b2648511ba5d8eeb81a615a1378db574fb96cd4e7 not found: ID does not exist" containerID="742a8e6935792d5e4a8aa63b2648511ba5d8eeb81a615a1378db574fb96cd4e7" Oct 30 09:44:49.851117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.851044 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742a8e6935792d5e4a8aa63b2648511ba5d8eeb81a615a1378db574fb96cd4e7"} err="failed to get container status \"742a8e6935792d5e4a8aa63b2648511ba5d8eeb81a615a1378db574fb96cd4e7\": rpc error: code = NotFound desc = could not find container \"742a8e6935792d5e4a8aa63b2648511ba5d8eeb81a615a1378db574fb96cd4e7\": container with ID starting with 742a8e6935792d5e4a8aa63b2648511ba5d8eeb81a615a1378db574fb96cd4e7 not found: ID does not exist" Oct 30 09:44:49.885141 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.885096 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" podStartSLOduration=1.885082987 podStartE2EDuration="1.885082987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:44:49.882216837 +0000 UTC m=+3007.744922659" watchObservedRunningTime="2026-10-30 09:44:49.885082987 +0000 UTC m=+3007.747788847" Oct 30 09:44:49.926840 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.926810 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c"] Oct 30 09:44:49.930127 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:49.930100 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbrlj9c"] Oct 30 09:44:50.258519 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:50.258486 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6cf721-374c-4a4e-b31f-17a351c4adf3" path="/var/lib/kubelet/pods/bd6cf721-374c-4a4e-b31f-17a351c4adf3/volumes" Oct 30 09:44:50.848795 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:50.848761 5938 generic.go:356] "Generic (PLEG): container finished" podID="f08e92f2-4e98-4e05-936f-e11730ed863e" containerID="bde2c52a8bae69afd7dff15efe5fbca351274e76370a665397fb6c01431c8054" exitCode=0 Oct 30 09:44:50.848992 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:50.848814 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" event={"ID":"f08e92f2-4e98-4e05-936f-e11730ed863e","Type":"ContainerDied","Data":"bde2c52a8bae69afd7dff15efe5fbca351274e76370a665397fb6c01431c8054"} Oct 30 09:44:52.011859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:52.011833 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" Oct 30 09:44:52.145594 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:52.145549 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f08e92f2-4e98-4e05-936f-e11730ed863e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08e92f2-4e98-4e05-936f-e11730ed863e-maas-api-xks-ca\") pod \"f08e92f2-4e98-4e05-936f-e11730ed863e\" (UID: \"f08e92f2-4e98-4e05-936f-e11730ed863e\") " Oct 30 09:44:52.145769 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:52.145662 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08e92f2-4e98-4e05-936f-e11730ed863e-kube-api-access-jf5bd\" (UniqueName: \"kubernetes.io/projected/f08e92f2-4e98-4e05-936f-e11730ed863e-kube-api-access-jf5bd\") pod \"f08e92f2-4e98-4e05-936f-e11730ed863e\" (UID: \"f08e92f2-4e98-4e05-936f-e11730ed863e\") " Oct 30 09:44:52.145827 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:52.145767 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f08e92f2-4e98-4e05-936f-e11730ed863e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08e92f2-4e98-4e05-936f-e11730ed863e-maas-api-openshift-ca\") pod \"f08e92f2-4e98-4e05-936f-e11730ed863e\" (UID: \"f08e92f2-4e98-4e05-936f-e11730ed863e\") " Oct 30 09:44:52.146181 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:52.146150 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08e92f2-4e98-4e05-936f-e11730ed863e-maas-api-openshift-ca" pod "f08e92f2-4e98-4e05-936f-e11730ed863e" (UID: "f08e92f2-4e98-4e05-936f-e11730ed863e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:44:52.146488 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:52.146424 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08e92f2-4e98-4e05-936f-e11730ed863e-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:52.147901 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:52.147877 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08e92f2-4e98-4e05-936f-e11730ed863e-maas-api-xks-ca" pod "f08e92f2-4e98-4e05-936f-e11730ed863e" (UID: "f08e92f2-4e98-4e05-936f-e11730ed863e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:44:52.148010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:52.147986 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08e92f2-4e98-4e05-936f-e11730ed863e-kube-api-access-jf5bd" pod "f08e92f2-4e98-4e05-936f-e11730ed863e" (UID: "f08e92f2-4e98-4e05-936f-e11730ed863e"). InnerVolumeSpecName "kube-api-access-jf5bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:52.248121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:52.248079 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf5bd\" (UniqueName: \"kubernetes.io/projected/f08e92f2-4e98-4e05-936f-e11730ed863e-kube-api-access-jf5bd\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:52.248121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:52.248118 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08e92f2-4e98-4e05-936f-e11730ed863e-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:52.866055 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:52.866028 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" Oct 30 09:44:52.866236 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:52.866064 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl" event={"ID":"f08e92f2-4e98-4e05-936f-e11730ed863e","Type":"ContainerDied","Data":"56efe7f5a21c8346f06bf2e82fb6bcc52b004d4466d2a7fae51b2f8c205f29d9"} Oct 30 09:44:52.866236 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:52.866116 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56efe7f5a21c8346f06bf2e82fb6bcc52b004d4466d2a7fae51b2f8c205f29d9" Oct 30 09:44:53.027369 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.027328 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" podUID="8455d270-3802-4b0a-b8e7-1399edbba05d" containerName="payload-processing" containerID="cri-o://47859c48adba0c3e0c665c375c2a23ad74723f4b39ebcfe365a16e4bcb743c4b" gracePeriod=25 Oct 30 09:44:53.311279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.311254 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:53.465466 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.465387 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8455d270-3802-4b0a-b8e7-1399edbba05d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8455d270-3802-4b0a-b8e7-1399edbba05d-provider-credentials\") pod \"8455d270-3802-4b0a-b8e7-1399edbba05d\" (UID: \"8455d270-3802-4b0a-b8e7-1399edbba05d\") " Oct 30 09:44:53.465466 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.465446 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8455d270-3802-4b0a-b8e7-1399edbba05d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8455d270-3802-4b0a-b8e7-1399edbba05d-plugins-config-volume\") pod \"8455d270-3802-4b0a-b8e7-1399edbba05d\" (UID: \"8455d270-3802-4b0a-b8e7-1399edbba05d\") " Oct 30 09:44:53.465712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.465553 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8455d270-3802-4b0a-b8e7-1399edbba05d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8455d270-3802-4b0a-b8e7-1399edbba05d-routing-overlay-volume\") pod \"8455d270-3802-4b0a-b8e7-1399edbba05d\" (UID: \"8455d270-3802-4b0a-b8e7-1399edbba05d\") " Oct 30 09:44:53.465712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.465599 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8455d270-3802-4b0a-b8e7-1399edbba05d-kube-api-access-rc7wj\" (UniqueName: \"kubernetes.io/projected/8455d270-3802-4b0a-b8e7-1399edbba05d-kube-api-access-rc7wj\") pod \"8455d270-3802-4b0a-b8e7-1399edbba05d\" (UID: \"8455d270-3802-4b0a-b8e7-1399edbba05d\") " Oct 30 09:44:53.465918 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.465896 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8455d270-3802-4b0a-b8e7-1399edbba05d-plugins-config-volume" pod "8455d270-3802-4b0a-b8e7-1399edbba05d" (UID: "8455d270-3802-4b0a-b8e7-1399edbba05d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:44:53.465967 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.465928 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8455d270-3802-4b0a-b8e7-1399edbba05d-routing-overlay-volume" pod "8455d270-3802-4b0a-b8e7-1399edbba05d" (UID: "8455d270-3802-4b0a-b8e7-1399edbba05d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:44:53.466211 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.466187 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8455d270-3802-4b0a-b8e7-1399edbba05d-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:53.466211 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.466209 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8455d270-3802-4b0a-b8e7-1399edbba05d-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:53.468297 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.468268 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8455d270-3802-4b0a-b8e7-1399edbba05d-provider-credentials" pod "8455d270-3802-4b0a-b8e7-1399edbba05d" (UID: "8455d270-3802-4b0a-b8e7-1399edbba05d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:53.468387 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.468306 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8455d270-3802-4b0a-b8e7-1399edbba05d-kube-api-access-rc7wj" pod "8455d270-3802-4b0a-b8e7-1399edbba05d" (UID: "8455d270-3802-4b0a-b8e7-1399edbba05d"). InnerVolumeSpecName "kube-api-access-rc7wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:53.567905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.567868 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8455d270-3802-4b0a-b8e7-1399edbba05d-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:53.567905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.567902 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc7wj\" (UniqueName: \"kubernetes.io/projected/8455d270-3802-4b0a-b8e7-1399edbba05d-kube-api-access-rc7wj\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:44:53.874355 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.874320 5938 generic.go:356] "Generic (PLEG): container finished" podID="8455d270-3802-4b0a-b8e7-1399edbba05d" containerID="47859c48adba0c3e0c665c375c2a23ad74723f4b39ebcfe365a16e4bcb743c4b" exitCode=0 Oct 30 09:44:53.874543 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.874380 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" Oct 30 09:44:53.874543 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.874407 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" event={"ID":"8455d270-3802-4b0a-b8e7-1399edbba05d","Type":"ContainerDied","Data":"47859c48adba0c3e0c665c375c2a23ad74723f4b39ebcfe365a16e4bcb743c4b"} Oct 30 09:44:53.874543 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.874483 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf" event={"ID":"8455d270-3802-4b0a-b8e7-1399edbba05d","Type":"ContainerDied","Data":"bbc891665c921da28598a9ad2b89f535917b365114b6ac0e458b61c26be24963"} Oct 30 09:44:53.874543 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.874505 5938 scope.go:122] "RemoveContainer" containerID="47859c48adba0c3e0c665c375c2a23ad74723f4b39ebcfe365a16e4bcb743c4b" Oct 30 09:44:53.886032 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.886014 5938 scope.go:122] "RemoveContainer" containerID="47859c48adba0c3e0c665c375c2a23ad74723f4b39ebcfe365a16e4bcb743c4b" Oct 30 09:44:53.886299 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:44:53.886282 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47859c48adba0c3e0c665c375c2a23ad74723f4b39ebcfe365a16e4bcb743c4b\": container with ID starting with 47859c48adba0c3e0c665c375c2a23ad74723f4b39ebcfe365a16e4bcb743c4b not found: ID does not exist" containerID="47859c48adba0c3e0c665c375c2a23ad74723f4b39ebcfe365a16e4bcb743c4b" Oct 30 09:44:53.886342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.886307 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47859c48adba0c3e0c665c375c2a23ad74723f4b39ebcfe365a16e4bcb743c4b"} err="failed to get container status \"47859c48adba0c3e0c665c375c2a23ad74723f4b39ebcfe365a16e4bcb743c4b\": rpc error: code = NotFound desc = could not find container \"47859c48adba0c3e0c665c375c2a23ad74723f4b39ebcfe365a16e4bcb743c4b\": container with ID starting with 47859c48adba0c3e0c665c375c2a23ad74723f4b39ebcfe365a16e4bcb743c4b not found: ID does not exist" Oct 30 09:44:53.931353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.931327 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf"] Oct 30 09:44:53.944914 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:53.944889 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555c4dlf"] Oct 30 09:44:54.259491 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:44:54.259411 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8455d270-3802-4b0a-b8e7-1399edbba05d" path="/var/lib/kubelet/pods/8455d270-3802-4b0a-b8e7-1399edbba05d/volumes" Oct 30 09:45:00.146772 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.146739 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm"] Oct 30 09:45:00.148782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.148700 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8455d270-3802-4b0a-b8e7-1399edbba05d" containerName="payload-processing" Oct 30 09:45:00.149009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.148992 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8455d270-3802-4b0a-b8e7-1399edbba05d" containerName="payload-processing" Oct 30 09:45:00.149143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.149132 5938 cpu_manager.go:414] "containerMap: removing container" podUID="bd6cf721-374c-4a4e-b31f-17a351c4adf3" containerName="payload-processing" Oct 30 09:45:00.149219 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.149209 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6cf721-374c-4a4e-b31f-17a351c4adf3" containerName="payload-processing" Oct 30 09:45:00.149307 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.149296 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f08e92f2-4e98-4e05-936f-e11730ed863e" containerName="revoke-keys" Oct 30 09:45:00.149384 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.149373 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08e92f2-4e98-4e05-936f-e11730ed863e" containerName="revoke-keys" Oct 30 09:45:00.149683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.149667 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8455d270-3802-4b0a-b8e7-1399edbba05d" containerName="payload-processing" Oct 30 09:45:00.149812 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.149801 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6cf721-374c-4a4e-b31f-17a351c4adf3" containerName="payload-processing" Oct 30 09:45:00.150019 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.149985 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08e92f2-4e98-4e05-936f-e11730ed863e" containerName="revoke-keys" Oct 30 09:45:00.159762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.159735 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" Oct 30 09:45:00.162285 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.162264 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj"] Oct 30 09:45:00.177025 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.176909 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6"] Oct 30 09:45:00.177266 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.177006 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" Oct 30 09:45:00.195258 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.195230 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:45:00.239917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.227446 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8"] Oct 30 09:45:00.239917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.235852 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" Oct 30 09:45:00.255561 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.254954 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26"] Oct 30 09:45:00.255823 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.255801 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" Oct 30 09:45:00.290654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.274570 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97btn\" (UniqueName: \"kubernetes.io/projected/7b1877d3-aaac-4287-8450-d02cb2497d3f-kube-api-access-97btn\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8\" (UID: \"7b1877d3-aaac-4287-8450-d02cb2497d3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" Oct 30 09:45:00.290654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.274681 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px48k\" (UniqueName: \"kubernetes.io/projected/ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4-kube-api-access-px48k\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj\" (UID: \"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" Oct 30 09:45:00.290654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.274742 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1877d3-aaac-4287-8450-d02cb2497d3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8\" (UID: \"7b1877d3-aaac-4287-8450-d02cb2497d3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" Oct 30 09:45:00.290654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.274834 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj\" (UID: \"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" Oct 30 09:45:00.290654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.274872 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kls2c\" (UniqueName: \"kubernetes.io/projected/5de510d4-441f-4aa1-98d7-c60848cd26ed-kube-api-access-kls2c\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6\" (UID: \"5de510d4-441f-4aa1-98d7-c60848cd26ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" Oct 30 09:45:00.293697 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.293662 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqkz2\" (UniqueName: \"kubernetes.io/projected/afc6a7ef-c174-4816-938d-817e57691e6b-kube-api-access-dqkz2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm\" (UID: \"afc6a7ef-c174-4816-938d-817e57691e6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" Oct 30 09:45:00.293846 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.293795 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de510d4-441f-4aa1-98d7-c60848cd26ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6\" (UID: \"5de510d4-441f-4aa1-98d7-c60848cd26ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" Oct 30 09:45:00.293908 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.293875 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc6a7ef-c174-4816-938d-817e57691e6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm\" (UID: \"afc6a7ef-c174-4816-938d-817e57691e6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" Oct 30 09:45:00.301020 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.300998 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" Oct 30 09:45:00.326601 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.326551 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm"] Oct 30 09:45:00.326762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.326639 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj"] Oct 30 09:45:00.326762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.326669 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6"] Oct 30 09:45:00.326762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.326698 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp"] Oct 30 09:45:00.333821 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.333797 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm"] Oct 30 09:45:00.339773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.334180 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" Oct 30 09:45:00.340847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.340831 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8"] Oct 30 09:45:00.341058 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.341044 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" Oct 30 09:45:00.341177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.341053 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz"] Oct 30 09:45:00.346864 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.346848 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm"] Oct 30 09:45:00.346995 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.346885 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp"] Oct 30 09:45:00.347065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.347011 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" Oct 30 09:45:00.348793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.348762 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn"] Oct 30 09:45:00.358054 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.358033 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz"] Oct 30 09:45:00.358172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.358073 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd"] Oct 30 09:45:00.358475 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.358352 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" Oct 30 09:45:00.368882 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.368865 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn"] Oct 30 09:45:00.369070 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.369060 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" Oct 30 09:45:00.375023 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.374709 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26"] Oct 30 09:45:00.380167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.379464 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd"] Oct 30 09:45:00.395692 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.395656 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqkz2\" (UniqueName: \"kubernetes.io/projected/afc6a7ef-c174-4816-938d-817e57691e6b-kube-api-access-dqkz2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm\" (UID: \"afc6a7ef-c174-4816-938d-817e57691e6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" Oct 30 09:45:00.395826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.395759 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f30e90f0-9862-48cc-8171-df2f58695096-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26\" (UID: \"f30e90f0-9862-48cc-8171-df2f58695096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" Oct 30 09:45:00.395826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.395816 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de510d4-441f-4aa1-98d7-c60848cd26ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6\" (UID: \"5de510d4-441f-4aa1-98d7-c60848cd26ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" Oct 30 09:45:00.395936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.395866 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd723952-1a2b-46d6-96b8-e19056165e36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn\" (UID: \"cd723952-1a2b-46d6-96b8-e19056165e36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" Oct 30 09:45:00.395936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.395915 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc6a7ef-c174-4816-938d-817e57691e6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm\" (UID: \"afc6a7ef-c174-4816-938d-817e57691e6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" Oct 30 09:45:00.396184 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.395953 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6ll8\" (UniqueName: \"kubernetes.io/projected/41e9f62f-6ced-4207-a09b-b820f0ad142f-kube-api-access-c6ll8\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz\" (UID: \"41e9f62f-6ced-4207-a09b-b820f0ad142f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" Oct 30 09:45:00.396184 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.396012 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e9f62f-6ced-4207-a09b-b820f0ad142f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz\" (UID: \"41e9f62f-6ced-4207-a09b-b820f0ad142f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" Oct 30 09:45:00.396184 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.396078 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmhff\" (UniqueName: \"kubernetes.io/projected/fc5fe483-3456-4b2f-b626-a14bb00dac59-kube-api-access-gmhff\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm\" (UID: \"fc5fe483-3456-4b2f-b626-a14bb00dac59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" Oct 30 09:45:00.396184 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.396119 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97btn\" (UniqueName: \"kubernetes.io/projected/7b1877d3-aaac-4287-8450-d02cb2497d3f-kube-api-access-97btn\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8\" (UID: \"7b1877d3-aaac-4287-8450-d02cb2497d3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" Oct 30 09:45:00.396184 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.396152 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l6nz\" (UniqueName: \"kubernetes.io/projected/cd723952-1a2b-46d6-96b8-e19056165e36-kube-api-access-2l6nz\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn\" (UID: \"cd723952-1a2b-46d6-96b8-e19056165e36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" Oct 30 09:45:00.396465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.396197 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40975fdb-ec84-40e3-8b8f-68bfd44c8123-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889225-fb6zp\" (UID: \"40975fdb-ec84-40e3-8b8f-68bfd44c8123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" Oct 30 09:45:00.396465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.396254 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px48k\" (UniqueName: \"kubernetes.io/projected/ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4-kube-api-access-px48k\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj\" (UID: \"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" Oct 30 09:45:00.396647 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.396626 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de510d4-441f-4aa1-98d7-c60848cd26ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6\" (UID: \"5de510d4-441f-4aa1-98d7-c60848cd26ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" Oct 30 09:45:00.396857 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.396812 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1877d3-aaac-4287-8450-d02cb2497d3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8\" (UID: \"7b1877d3-aaac-4287-8450-d02cb2497d3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" Oct 30 09:45:00.396939 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.396866 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch896\" (UniqueName: \"kubernetes.io/projected/40975fdb-ec84-40e3-8b8f-68bfd44c8123-kube-api-access-ch896\") pod \"maas-api-key-cleanup-29889225-fb6zp\" (UID: \"40975fdb-ec84-40e3-8b8f-68bfd44c8123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" Oct 30 09:45:00.396939 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.396909 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc5fe483-3456-4b2f-b626-a14bb00dac59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm\" (UID: \"fc5fe483-3456-4b2f-b626-a14bb00dac59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" Oct 30 09:45:00.397017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.396996 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj\" (UID: \"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" Oct 30 09:45:00.397059 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.397034 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kls2c\" (UniqueName: \"kubernetes.io/projected/5de510d4-441f-4aa1-98d7-c60848cd26ed-kube-api-access-kls2c\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6\" (UID: \"5de510d4-441f-4aa1-98d7-c60848cd26ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" Oct 30 09:45:00.397096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.397084 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4gcf\" (UniqueName: \"kubernetes.io/projected/f30e90f0-9862-48cc-8171-df2f58695096-kube-api-access-j4gcf\") pod \"maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26\" (UID: \"f30e90f0-9862-48cc-8171-df2f58695096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" Oct 30 09:45:00.397545 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.397348 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc6a7ef-c174-4816-938d-817e57691e6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm\" (UID: \"afc6a7ef-c174-4816-938d-817e57691e6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" Oct 30 09:45:00.397925 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.397902 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1877d3-aaac-4287-8450-d02cb2497d3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8\" (UID: \"7b1877d3-aaac-4287-8450-d02cb2497d3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" Oct 30 09:45:00.398064 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.398045 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj\" (UID: \"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" Oct 30 09:45:00.419832 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.419751 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px48k\" (UniqueName: \"kubernetes.io/projected/ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4-kube-api-access-px48k\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj\" (UID: \"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" Oct 30 09:45:00.426562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.426530 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kls2c\" (UniqueName: \"kubernetes.io/projected/5de510d4-441f-4aa1-98d7-c60848cd26ed-kube-api-access-kls2c\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6\" (UID: \"5de510d4-441f-4aa1-98d7-c60848cd26ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" Oct 30 09:45:00.426740 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.426530 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97btn\" (UniqueName: \"kubernetes.io/projected/7b1877d3-aaac-4287-8450-d02cb2497d3f-kube-api-access-97btn\") pod \"maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8\" (UID: \"7b1877d3-aaac-4287-8450-d02cb2497d3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" Oct 30 09:45:00.429193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.429158 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqkz2\" (UniqueName: \"kubernetes.io/projected/afc6a7ef-c174-4816-938d-817e57691e6b-kube-api-access-dqkz2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm\" (UID: \"afc6a7ef-c174-4816-938d-817e57691e6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" Oct 30 09:45:00.498767 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.498729 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch896\" (UniqueName: \"kubernetes.io/projected/40975fdb-ec84-40e3-8b8f-68bfd44c8123-kube-api-access-ch896\") pod \"maas-api-key-cleanup-29889225-fb6zp\" (UID: \"40975fdb-ec84-40e3-8b8f-68bfd44c8123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" Oct 30 09:45:00.498957 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.498774 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc5fe483-3456-4b2f-b626-a14bb00dac59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm\" (UID: \"fc5fe483-3456-4b2f-b626-a14bb00dac59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" Oct 30 09:45:00.498957 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.498861 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gcf\" (UniqueName: \"kubernetes.io/projected/f30e90f0-9862-48cc-8171-df2f58695096-kube-api-access-j4gcf\") pod \"maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26\" (UID: \"f30e90f0-9862-48cc-8171-df2f58695096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" Oct 30 09:45:00.498957 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.498906 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2qmf\" (UniqueName: \"kubernetes.io/projected/ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4-kube-api-access-n2qmf\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd\" (UID: \"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" Oct 30 09:45:00.499147 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.499118 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f30e90f0-9862-48cc-8171-df2f58695096-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26\" (UID: \"f30e90f0-9862-48cc-8171-df2f58695096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" Oct 30 09:45:00.499220 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.499202 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd723952-1a2b-46d6-96b8-e19056165e36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn\" (UID: \"cd723952-1a2b-46d6-96b8-e19056165e36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" Oct 30 09:45:00.499279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.499263 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6ll8\" (UniqueName: \"kubernetes.io/projected/41e9f62f-6ced-4207-a09b-b820f0ad142f-kube-api-access-c6ll8\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz\" (UID: \"41e9f62f-6ced-4207-a09b-b820f0ad142f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" Oct 30 09:45:00.499334 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.499296 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e9f62f-6ced-4207-a09b-b820f0ad142f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz\" (UID: \"41e9f62f-6ced-4207-a09b-b820f0ad142f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" Oct 30 09:45:00.499393 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.499332 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd\" (UID: \"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" Oct 30 09:45:00.499393 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.499379 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmhff\" (UniqueName: \"kubernetes.io/projected/fc5fe483-3456-4b2f-b626-a14bb00dac59-kube-api-access-gmhff\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm\" (UID: \"fc5fe483-3456-4b2f-b626-a14bb00dac59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" Oct 30 09:45:00.499500 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.499438 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l6nz\" (UniqueName: \"kubernetes.io/projected/cd723952-1a2b-46d6-96b8-e19056165e36-kube-api-access-2l6nz\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn\" (UID: \"cd723952-1a2b-46d6-96b8-e19056165e36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" Oct 30 09:45:00.499500 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.499492 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40975fdb-ec84-40e3-8b8f-68bfd44c8123-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889225-fb6zp\" (UID: \"40975fdb-ec84-40e3-8b8f-68bfd44c8123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" Oct 30 09:45:00.499646 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.499523 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc5fe483-3456-4b2f-b626-a14bb00dac59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm\" (UID: \"fc5fe483-3456-4b2f-b626-a14bb00dac59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" Oct 30 09:45:00.499793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.499766 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f30e90f0-9862-48cc-8171-df2f58695096-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26\" (UID: \"f30e90f0-9862-48cc-8171-df2f58695096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" Oct 30 09:45:00.500299 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.500272 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e9f62f-6ced-4207-a09b-b820f0ad142f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz\" (UID: \"41e9f62f-6ced-4207-a09b-b820f0ad142f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" Oct 30 09:45:00.500402 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.500305 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd723952-1a2b-46d6-96b8-e19056165e36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn\" (UID: \"cd723952-1a2b-46d6-96b8-e19056165e36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" Oct 30 09:45:00.500402 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.500367 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40975fdb-ec84-40e3-8b8f-68bfd44c8123-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889225-fb6zp\" (UID: \"40975fdb-ec84-40e3-8b8f-68bfd44c8123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" Oct 30 09:45:00.516746 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.516711 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" Oct 30 09:45:00.521364 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.521343 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gcf\" (UniqueName: \"kubernetes.io/projected/f30e90f0-9862-48cc-8171-df2f58695096-kube-api-access-j4gcf\") pod \"maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26\" (UID: \"f30e90f0-9862-48cc-8171-df2f58695096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" Oct 30 09:45:00.521423 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.521345 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l6nz\" (UniqueName: \"kubernetes.io/projected/cd723952-1a2b-46d6-96b8-e19056165e36-kube-api-access-2l6nz\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn\" (UID: \"cd723952-1a2b-46d6-96b8-e19056165e36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" Oct 30 09:45:00.521652 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.521631 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch896\" (UniqueName: \"kubernetes.io/projected/40975fdb-ec84-40e3-8b8f-68bfd44c8123-kube-api-access-ch896\") pod \"maas-api-key-cleanup-29889225-fb6zp\" (UID: \"40975fdb-ec84-40e3-8b8f-68bfd44c8123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" Oct 30 09:45:00.522330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.522309 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6ll8\" (UniqueName: \"kubernetes.io/projected/41e9f62f-6ced-4207-a09b-b820f0ad142f-kube-api-access-c6ll8\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz\" (UID: \"41e9f62f-6ced-4207-a09b-b820f0ad142f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" Oct 30 09:45:00.523054 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.523034 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmhff\" (UniqueName: \"kubernetes.io/projected/fc5fe483-3456-4b2f-b626-a14bb00dac59-kube-api-access-gmhff\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm\" (UID: \"fc5fe483-3456-4b2f-b626-a14bb00dac59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" Oct 30 09:45:00.557321 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.557289 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" Oct 30 09:45:00.569823 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.569799 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" Oct 30 09:45:00.601308 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.601266 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2qmf\" (UniqueName: \"kubernetes.io/projected/ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4-kube-api-access-n2qmf\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd\" (UID: \"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" Oct 30 09:45:00.601721 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.601700 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd\" (UID: \"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" Oct 30 09:45:00.602655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.602634 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd\" (UID: \"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" Oct 30 09:45:00.608356 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.608322 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" Oct 30 09:45:00.643041 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.642900 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" Oct 30 09:45:00.665100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.665065 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" Oct 30 09:45:00.666291 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.666258 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2qmf\" (UniqueName: \"kubernetes.io/projected/ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4-kube-api-access-n2qmf\") pod \"maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd\" (UID: \"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" Oct 30 09:45:00.684052 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.683848 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" Oct 30 09:45:00.701684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.695818 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" Oct 30 09:45:00.725201 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.725167 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" Oct 30 09:45:00.774540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.774295 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" Oct 30 09:45:00.865990 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:45:00.864971 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc6a7ef_c174_4816_938d_817e57691e6b.slice/crio-bbec99ecb0fd26ef8e3439f17e6c0d3cc9f165bddc703a7d63463f327a0476e3 WatchSource:0}: Error finding container bbec99ecb0fd26ef8e3439f17e6c0d3cc9f165bddc703a7d63463f327a0476e3: Status 404 returned error can't find the container with id bbec99ecb0fd26ef8e3439f17e6c0d3cc9f165bddc703a7d63463f327a0476e3 Oct 30 09:45:00.878878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.878832 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm"] Oct 30 09:45:00.961686 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.961520 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" event={"ID":"afc6a7ef-c174-4816-938d-817e57691e6b","Type":"ContainerStarted","Data":"bbec99ecb0fd26ef8e3439f17e6c0d3cc9f165bddc703a7d63463f327a0476e3"} Oct 30 09:45:00.966846 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:00.966743 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj"] Oct 30 09:45:01.009518 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.009445 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6"] Oct 30 09:45:01.012954 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:45:01.012918 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de510d4_441f_4aa1_98d7_c60848cd26ed.slice/crio-4eaeef0f9af36561942213cda6120ccaaa99764b766d24e867341fcb77c60602 WatchSource:0}: Error finding container 4eaeef0f9af36561942213cda6120ccaaa99764b766d24e867341fcb77c60602: Status 404 returned error can't find the container with id 4eaeef0f9af36561942213cda6120ccaaa99764b766d24e867341fcb77c60602 Oct 30 09:45:01.234420 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:45:01.234388 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1877d3_aaac_4287_8450_d02cb2497d3f.slice/crio-671c616185f9fd2c35d2b5234b740554af2ebeecc5a3f55dedc5c4892513ca21 WatchSource:0}: Error finding container 671c616185f9fd2c35d2b5234b740554af2ebeecc5a3f55dedc5c4892513ca21: Status 404 returned error can't find the container with id 671c616185f9fd2c35d2b5234b740554af2ebeecc5a3f55dedc5c4892513ca21 Oct 30 09:45:01.235016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.234963 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8"] Oct 30 09:45:01.246103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.246066 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz"] Oct 30 09:45:01.298700 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.298673 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26"] Oct 30 09:45:01.298956 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:45:01.298929 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30e90f0_9862_48cc_8171_df2f58695096.slice/crio-2b2ccd5d2a87e1433b3c867f51aca0865e15959253d4496accbdbf7763ebeb5e WatchSource:0}: Error finding container 2b2ccd5d2a87e1433b3c867f51aca0865e15959253d4496accbdbf7763ebeb5e: Status 404 returned error can't find the container with id 2b2ccd5d2a87e1433b3c867f51aca0865e15959253d4496accbdbf7763ebeb5e Oct 30 09:45:01.326839 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.326798 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm"] Oct 30 09:45:01.359847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.359824 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp"] Oct 30 09:45:01.368120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.368099 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd"] Oct 30 09:45:01.413369 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.413344 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn"] Oct 30 09:45:01.424178 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:45:01.424151 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5fe483_3456_4b2f_b626_a14bb00dac59.slice/crio-81defbe9aa9774b76eb7d51d596b52578cb64e3d9e7148a11e3502cd6e5dec30 WatchSource:0}: Error finding container 81defbe9aa9774b76eb7d51d596b52578cb64e3d9e7148a11e3502cd6e5dec30: Status 404 returned error can't find the container with id 81defbe9aa9774b76eb7d51d596b52578cb64e3d9e7148a11e3502cd6e5dec30 Oct 30 09:45:01.523776 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:45:01.523674 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40975fdb_ec84_40e3_8b8f_68bfd44c8123.slice/crio-2b71a9fa5dd04a76434a370c490e9c3698cc2a0383c75ab93ba71119a7bf5994 WatchSource:0}: Error finding container 2b71a9fa5dd04a76434a370c490e9c3698cc2a0383c75ab93ba71119a7bf5994: Status 404 returned error can't find the container with id 2b71a9fa5dd04a76434a370c490e9c3698cc2a0383c75ab93ba71119a7bf5994 Oct 30 09:45:01.622416 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:45:01.622378 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec402ea4_3c3b_4d53_8ffb_aa7443d0a2c4.slice/crio-3ea03e7521dee0d47bfa668f4ff3171f63fa12836b9a903ac51403079e793cdb WatchSource:0}: Error finding container 3ea03e7521dee0d47bfa668f4ff3171f63fa12836b9a903ac51403079e793cdb: Status 404 returned error can't find the container with id 3ea03e7521dee0d47bfa668f4ff3171f63fa12836b9a903ac51403079e793cdb Oct 30 09:45:01.624323 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:45:01.624299 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd723952_1a2b_46d6_96b8_e19056165e36.slice/crio-5e8bde6700cc26f3de152588d82a28c1e4c8d9fa5c79bf1fa9ee3bf31790d70c WatchSource:0}: Error finding container 5e8bde6700cc26f3de152588d82a28c1e4c8d9fa5c79bf1fa9ee3bf31790d70c: Status 404 returned error can't find the container with id 5e8bde6700cc26f3de152588d82a28c1e4c8d9fa5c79bf1fa9ee3bf31790d70c Oct 30 09:45:01.970455 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.970417 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" event={"ID":"afc6a7ef-c174-4816-938d-817e57691e6b","Type":"ContainerStarted","Data":"6c57dc3fb4f9019742c9c9373b51226adabc13123b6acc6c2d8e18aa5f8a0db1"} Oct 30 09:45:01.972379 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.972351 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" event={"ID":"f30e90f0-9862-48cc-8171-df2f58695096","Type":"ContainerStarted","Data":"f417a8d2a5ab012b0cf10fbf484accb7f0ed8d56638ff940ed3cd2dd9479af4a"} Oct 30 09:45:01.972510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.972404 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" event={"ID":"f30e90f0-9862-48cc-8171-df2f58695096","Type":"ContainerStarted","Data":"2b2ccd5d2a87e1433b3c867f51aca0865e15959253d4496accbdbf7763ebeb5e"} Oct 30 09:45:01.974049 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.974027 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" event={"ID":"7b1877d3-aaac-4287-8450-d02cb2497d3f","Type":"ContainerStarted","Data":"060cc4a01969a3afe2c09bd43a452ea6441084c24607b6caf9dc91ae74f5b669"} Oct 30 09:45:01.974159 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.974069 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" event={"ID":"7b1877d3-aaac-4287-8450-d02cb2497d3f","Type":"ContainerStarted","Data":"671c616185f9fd2c35d2b5234b740554af2ebeecc5a3f55dedc5c4892513ca21"} Oct 30 09:45:01.975856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.975833 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" event={"ID":"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4","Type":"ContainerStarted","Data":"65a372ab142ae68913d0ef8540926f98bb71f4558cab3d4ebf065266e75ad2f0"} Oct 30 09:45:01.975962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.975877 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" event={"ID":"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4","Type":"ContainerStarted","Data":"d1f2b85ba1714210d54bc1292271768ab142b3438b67926c491ac72c71b86936"} Oct 30 09:45:01.977257 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.977233 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" event={"ID":"cd723952-1a2b-46d6-96b8-e19056165e36","Type":"ContainerStarted","Data":"5e8bde6700cc26f3de152588d82a28c1e4c8d9fa5c79bf1fa9ee3bf31790d70c"} Oct 30 09:45:01.978630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.978609 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" event={"ID":"40975fdb-ec84-40e3-8b8f-68bfd44c8123","Type":"ContainerStarted","Data":"2b71a9fa5dd04a76434a370c490e9c3698cc2a0383c75ab93ba71119a7bf5994"} Oct 30 09:45:01.979969 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.979945 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" event={"ID":"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4","Type":"ContainerStarted","Data":"3ea03e7521dee0d47bfa668f4ff3171f63fa12836b9a903ac51403079e793cdb"} Oct 30 09:45:01.981415 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.981393 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" event={"ID":"41e9f62f-6ced-4207-a09b-b820f0ad142f","Type":"ContainerStarted","Data":"f5e68c68353796d7d6bc791a30088c9e9dd6f1c95a10c333bb98df48ef38be4a"} Oct 30 09:45:01.981495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.981435 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" event={"ID":"41e9f62f-6ced-4207-a09b-b820f0ad142f","Type":"ContainerStarted","Data":"8eccb25ac8dcc39270ca37e42e5611f64aedbae9af0d635801a30e2bd1d618dc"} Oct 30 09:45:01.982631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.982610 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" event={"ID":"fc5fe483-3456-4b2f-b626-a14bb00dac59","Type":"ContainerStarted","Data":"81defbe9aa9774b76eb7d51d596b52578cb64e3d9e7148a11e3502cd6e5dec30"} Oct 30 09:45:01.984103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.984084 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" event={"ID":"5de510d4-441f-4aa1-98d7-c60848cd26ed","Type":"ContainerStarted","Data":"cfa8f4a30c608ac82c6d08d1c1f7026e1ea96234746eab50e3ad4915416c8e7a"} Oct 30 09:45:01.984177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:01.984123 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" event={"ID":"5de510d4-441f-4aa1-98d7-c60848cd26ed","Type":"ContainerStarted","Data":"4eaeef0f9af36561942213cda6120ccaaa99764b766d24e867341fcb77c60602"} Oct 30 09:45:02.013034 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:02.012988 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" podStartSLOduration=2.012975114 podStartE2EDuration="2.012975114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:02.012576318 +0000 UTC m=+3019.875282140" watchObservedRunningTime="2026-10-30 09:45:02.012975114 +0000 UTC m=+3019.875680935" Oct 30 09:45:02.054006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:02.053963 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" podStartSLOduration=2.053945534 podStartE2EDuration="2.053945534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:02.051364987 +0000 UTC m=+3019.914070807" watchObservedRunningTime="2026-10-30 09:45:02.053945534 +0000 UTC m=+3019.916651359" Oct 30 09:45:02.095324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:02.095273 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" podStartSLOduration=2.095257094 podStartE2EDuration="2.095257094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:02.090190482 +0000 UTC m=+3019.952896338" watchObservedRunningTime="2026-10-30 09:45:02.095257094 +0000 UTC m=+3019.957962921" Oct 30 09:45:02.131806 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:02.131747 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" podStartSLOduration=2.131721875 podStartE2EDuration="2.131721875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:02.129498775 +0000 UTC m=+3019.992204599" watchObservedRunningTime="2026-10-30 09:45:02.131721875 +0000 UTC m=+3019.994427698" Oct 30 09:45:02.171680 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:02.171627 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" podStartSLOduration=2.171570622 podStartE2EDuration="2.171570622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:02.16854583 +0000 UTC m=+3020.031251651" watchObservedRunningTime="2026-10-30 09:45:02.171570622 +0000 UTC m=+3020.034276444" Oct 30 09:45:02.209850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:02.209808 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" podStartSLOduration=2.209795279 podStartE2EDuration="2.209795279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:02.208109476 +0000 UTC m=+3020.070815297" watchObservedRunningTime="2026-10-30 09:45:02.209795279 +0000 UTC m=+3020.072501102" Oct 30 09:45:02.993724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:02.993694 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" event={"ID":"40975fdb-ec84-40e3-8b8f-68bfd44c8123","Type":"ContainerStarted","Data":"262e118ea5d74bd20eecedd54a2833701efa8e071c6079fe39b2384648d46d39"} Oct 30 09:45:02.995434 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:02.995408 5938 generic.go:356] "Generic (PLEG): container finished" podID="ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" containerID="a1f718acbed772241777361975ce0b32965c8a542f116efaf623753433cd4904" exitCode=60 Oct 30 09:45:02.995577 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:02.995488 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" event={"ID":"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4","Type":"ContainerDied","Data":"a1f718acbed772241777361975ce0b32965c8a542f116efaf623753433cd4904"} Oct 30 09:45:02.995793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:02.995775 5938 scope.go:122] "RemoveContainer" containerID="a1f718acbed772241777361975ce0b32965c8a542f116efaf623753433cd4904" Oct 30 09:45:02.997232 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:02.997211 5938 generic.go:356] "Generic (PLEG): container finished" podID="41e9f62f-6ced-4207-a09b-b820f0ad142f" containerID="f5e68c68353796d7d6bc791a30088c9e9dd6f1c95a10c333bb98df48ef38be4a" exitCode=60 Oct 30 09:45:02.997356 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:02.997292 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" event={"ID":"41e9f62f-6ced-4207-a09b-b820f0ad142f","Type":"ContainerDied","Data":"f5e68c68353796d7d6bc791a30088c9e9dd6f1c95a10c333bb98df48ef38be4a"} Oct 30 09:45:02.997576 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:02.997529 5938 scope.go:122] "RemoveContainer" containerID="f5e68c68353796d7d6bc791a30088c9e9dd6f1c95a10c333bb98df48ef38be4a" Oct 30 09:45:02.999164 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:02.999141 5938 generic.go:356] "Generic (PLEG): container finished" podID="fc5fe483-3456-4b2f-b626-a14bb00dac59" containerID="0b40a6b997e1ca8b35d79eb9ba046b4b42adbd5eb28750a75c4cb3f98ef230c3" exitCode=60 Oct 30 09:45:02.999271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:02.999230 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" event={"ID":"fc5fe483-3456-4b2f-b626-a14bb00dac59","Type":"ContainerDied","Data":"0b40a6b997e1ca8b35d79eb9ba046b4b42adbd5eb28750a75c4cb3f98ef230c3"} Oct 30 09:45:02.999506 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:02.999455 5938 scope.go:122] "RemoveContainer" containerID="0b40a6b997e1ca8b35d79eb9ba046b4b42adbd5eb28750a75c4cb3f98ef230c3" Oct 30 09:45:03.001207 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.001148 5938 generic.go:356] "Generic (PLEG): container finished" podID="5de510d4-441f-4aa1-98d7-c60848cd26ed" containerID="cfa8f4a30c608ac82c6d08d1c1f7026e1ea96234746eab50e3ad4915416c8e7a" exitCode=60 Oct 30 09:45:03.001344 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.001321 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" event={"ID":"5de510d4-441f-4aa1-98d7-c60848cd26ed","Type":"ContainerDied","Data":"cfa8f4a30c608ac82c6d08d1c1f7026e1ea96234746eab50e3ad4915416c8e7a"} Oct 30 09:45:03.001621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.001572 5938 scope.go:122] "RemoveContainer" containerID="cfa8f4a30c608ac82c6d08d1c1f7026e1ea96234746eab50e3ad4915416c8e7a" Oct 30 09:45:03.003047 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.003025 5938 generic.go:356] "Generic (PLEG): container finished" podID="afc6a7ef-c174-4816-938d-817e57691e6b" containerID="6c57dc3fb4f9019742c9c9373b51226adabc13123b6acc6c2d8e18aa5f8a0db1" exitCode=60 Oct 30 09:45:03.003162 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.003102 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" event={"ID":"afc6a7ef-c174-4816-938d-817e57691e6b","Type":"ContainerDied","Data":"6c57dc3fb4f9019742c9c9373b51226adabc13123b6acc6c2d8e18aa5f8a0db1"} Oct 30 09:45:03.003293 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.003276 5938 scope.go:122] "RemoveContainer" containerID="6c57dc3fb4f9019742c9c9373b51226adabc13123b6acc6c2d8e18aa5f8a0db1" Oct 30 09:45:03.004689 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.004665 5938 generic.go:356] "Generic (PLEG): container finished" podID="f30e90f0-9862-48cc-8171-df2f58695096" containerID="f417a8d2a5ab012b0cf10fbf484accb7f0ed8d56638ff940ed3cd2dd9479af4a" exitCode=60 Oct 30 09:45:03.004778 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.004694 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" event={"ID":"f30e90f0-9862-48cc-8171-df2f58695096","Type":"ContainerDied","Data":"f417a8d2a5ab012b0cf10fbf484accb7f0ed8d56638ff940ed3cd2dd9479af4a"} Oct 30 09:45:03.004917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.004904 5938 scope.go:122] "RemoveContainer" containerID="f417a8d2a5ab012b0cf10fbf484accb7f0ed8d56638ff940ed3cd2dd9479af4a" Oct 30 09:45:03.006302 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.006283 5938 generic.go:356] "Generic (PLEG): container finished" podID="7b1877d3-aaac-4287-8450-d02cb2497d3f" containerID="060cc4a01969a3afe2c09bd43a452ea6441084c24607b6caf9dc91ae74f5b669" exitCode=60 Oct 30 09:45:03.006394 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.006343 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" event={"ID":"7b1877d3-aaac-4287-8450-d02cb2497d3f","Type":"ContainerDied","Data":"060cc4a01969a3afe2c09bd43a452ea6441084c24607b6caf9dc91ae74f5b669"} Oct 30 09:45:03.006616 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.006579 5938 scope.go:122] "RemoveContainer" containerID="060cc4a01969a3afe2c09bd43a452ea6441084c24607b6caf9dc91ae74f5b669" Oct 30 09:45:03.041442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.040209 5938 generic.go:356] "Generic (PLEG): container finished" podID="ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" containerID="65a372ab142ae68913d0ef8540926f98bb71f4558cab3d4ebf065266e75ad2f0" exitCode=60 Oct 30 09:45:03.041442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.040338 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" event={"ID":"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4","Type":"ContainerDied","Data":"65a372ab142ae68913d0ef8540926f98bb71f4558cab3d4ebf065266e75ad2f0"} Oct 30 09:45:03.041442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.040880 5938 scope.go:122] "RemoveContainer" containerID="65a372ab142ae68913d0ef8540926f98bb71f4558cab3d4ebf065266e75ad2f0" Oct 30 09:45:03.044683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.044656 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" event={"ID":"cd723952-1a2b-46d6-96b8-e19056165e36","Type":"ContainerStarted","Data":"be4ab14602642c5327201bec43baeacbbc2fd64d7fbde91c63fc9ed2ed13e55f"} Oct 30 09:45:03.044960 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.044908 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" podStartSLOduration=3.044891077 podStartE2EDuration="3.044891077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:03.037202157 +0000 UTC m=+3020.899907990" watchObservedRunningTime="2026-10-30 09:45:03.044891077 +0000 UTC m=+3020.907596901" Oct 30 09:45:03.045097 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:03.045038 5938 scope.go:122] "RemoveContainer" containerID="be4ab14602642c5327201bec43baeacbbc2fd64d7fbde91c63fc9ed2ed13e55f" Oct 30 09:45:04.054347 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.054312 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" event={"ID":"afc6a7ef-c174-4816-938d-817e57691e6b","Type":"ContainerStarted","Data":"fd62ffce0490dce7be36d758653f245238a8f909f79c3d17482bfc1f582a02d0"} Oct 30 09:45:04.056388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.056361 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" event={"ID":"f30e90f0-9862-48cc-8171-df2f58695096","Type":"ContainerStarted","Data":"ea06a40bf02e363db203aef2556942f3efedc74fdd574dc7758ea3ffeefc3567"} Oct 30 09:45:04.058339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.058314 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" event={"ID":"7b1877d3-aaac-4287-8450-d02cb2497d3f","Type":"ContainerStarted","Data":"8353539443f44a6918c8aedf958602f16accb42bd7cd7e090a48eea90f0bd793"} Oct 30 09:45:04.060227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.060201 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" event={"ID":"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4","Type":"ContainerStarted","Data":"5d57dc0345124fa1885d90fbebf30d711a4ac42281200077f2bbe6467a6df0b7"} Oct 30 09:45:04.062018 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.061993 5938 generic.go:356] "Generic (PLEG): container finished" podID="cd723952-1a2b-46d6-96b8-e19056165e36" containerID="be4ab14602642c5327201bec43baeacbbc2fd64d7fbde91c63fc9ed2ed13e55f" exitCode=60 Oct 30 09:45:04.062157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.062074 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" event={"ID":"cd723952-1a2b-46d6-96b8-e19056165e36","Type":"ContainerDied","Data":"be4ab14602642c5327201bec43baeacbbc2fd64d7fbde91c63fc9ed2ed13e55f"} Oct 30 09:45:04.062157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.062114 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" event={"ID":"cd723952-1a2b-46d6-96b8-e19056165e36","Type":"ContainerStarted","Data":"4bbc9edbe7c7d30d65aed6a3bf3cb942706cedcc0f0230b1fac90a8bbd1d4c9f"} Oct 30 09:45:04.063628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.063605 5938 generic.go:356] "Generic (PLEG): container finished" podID="40975fdb-ec84-40e3-8b8f-68bfd44c8123" containerID="262e118ea5d74bd20eecedd54a2833701efa8e071c6079fe39b2384648d46d39" exitCode=60 Oct 30 09:45:04.063735 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.063706 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" event={"ID":"40975fdb-ec84-40e3-8b8f-68bfd44c8123","Type":"ContainerDied","Data":"262e118ea5d74bd20eecedd54a2833701efa8e071c6079fe39b2384648d46d39"} Oct 30 09:45:04.063875 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.063859 5938 scope.go:122] "RemoveContainer" containerID="262e118ea5d74bd20eecedd54a2833701efa8e071c6079fe39b2384648d46d39" Oct 30 09:45:04.065728 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.065707 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" event={"ID":"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4","Type":"ContainerStarted","Data":"502c2cd7fdd2fa07623450fde9339a64ee9eada850242361b01a7606d2a3aa7c"} Oct 30 09:45:04.067771 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.067752 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" event={"ID":"41e9f62f-6ced-4207-a09b-b820f0ad142f","Type":"ContainerStarted","Data":"f7f6f46dfbe87df03e689146716ddcbc62058ba0bb793c1527e71b5b60e5099a"} Oct 30 09:45:04.069745 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.069724 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" event={"ID":"fc5fe483-3456-4b2f-b626-a14bb00dac59","Type":"ContainerStarted","Data":"15fb72aa6969e3a33e94a40765e5b3878210a81b04e04e71da4fce5ba4b2caf5"} Oct 30 09:45:04.071954 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.071929 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" event={"ID":"5de510d4-441f-4aa1-98d7-c60848cd26ed","Type":"ContainerStarted","Data":"3e5a57f44b785397994ce2cdee03b6b5d12d3c9d5a35dc29c5aea62196288801"} Oct 30 09:45:04.136087 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.136039 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" podStartSLOduration=4.136024777 podStartE2EDuration="4.136024777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:04.134455285 +0000 UTC m=+3021.997161112" watchObservedRunningTime="2026-10-30 09:45:04.136024777 +0000 UTC m=+3021.998730598" Oct 30 09:45:04.175159 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.175111 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" podStartSLOduration=4.175097216 podStartE2EDuration="4.175097216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:04.173532406 +0000 UTC m=+3022.036238226" watchObservedRunningTime="2026-10-30 09:45:04.175097216 +0000 UTC m=+3022.037803035" Oct 30 09:45:04.218055 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:04.217961 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" podStartSLOduration=4.217947634 podStartE2EDuration="4.217947634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:04.21450752 +0000 UTC m=+3022.077213345" watchObservedRunningTime="2026-10-30 09:45:04.217947634 +0000 UTC m=+3022.080653456" Oct 30 09:45:05.082318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.082280 5938 generic.go:356] "Generic (PLEG): container finished" podID="41e9f62f-6ced-4207-a09b-b820f0ad142f" containerID="f7f6f46dfbe87df03e689146716ddcbc62058ba0bb793c1527e71b5b60e5099a" exitCode=60 Oct 30 09:45:05.082808 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.082387 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" event={"ID":"41e9f62f-6ced-4207-a09b-b820f0ad142f","Type":"ContainerDied","Data":"f7f6f46dfbe87df03e689146716ddcbc62058ba0bb793c1527e71b5b60e5099a"} Oct 30 09:45:05.082808 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.082489 5938 scope.go:122] "RemoveContainer" containerID="f5e68c68353796d7d6bc791a30088c9e9dd6f1c95a10c333bb98df48ef38be4a" Oct 30 09:45:05.082980 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.082958 5938 scope.go:122] "RemoveContainer" containerID="f7f6f46dfbe87df03e689146716ddcbc62058ba0bb793c1527e71b5b60e5099a" Oct 30 09:45:05.083530 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:05.083369 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz_odh-ai-gateway-infra(41e9f62f-6ced-4207-a09b-b820f0ad142f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" podUID="41e9f62f-6ced-4207-a09b-b820f0ad142f" Oct 30 09:45:05.085621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.085570 5938 generic.go:356] "Generic (PLEG): container finished" podID="fc5fe483-3456-4b2f-b626-a14bb00dac59" containerID="15fb72aa6969e3a33e94a40765e5b3878210a81b04e04e71da4fce5ba4b2caf5" exitCode=60 Oct 30 09:45:05.085791 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.085621 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" event={"ID":"fc5fe483-3456-4b2f-b626-a14bb00dac59","Type":"ContainerDied","Data":"15fb72aa6969e3a33e94a40765e5b3878210a81b04e04e71da4fce5ba4b2caf5"} Oct 30 09:45:05.086354 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.086332 5938 scope.go:122] "RemoveContainer" containerID="15fb72aa6969e3a33e94a40765e5b3878210a81b04e04e71da4fce5ba4b2caf5" Oct 30 09:45:05.086797 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:05.086755 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm_odh-ai-gateway-infra(fc5fe483-3456-4b2f-b626-a14bb00dac59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" podUID="fc5fe483-3456-4b2f-b626-a14bb00dac59" Oct 30 09:45:05.088478 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.088452 5938 generic.go:356] "Generic (PLEG): container finished" podID="5de510d4-441f-4aa1-98d7-c60848cd26ed" containerID="3e5a57f44b785397994ce2cdee03b6b5d12d3c9d5a35dc29c5aea62196288801" exitCode=60 Oct 30 09:45:05.089163 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.089057 5938 scope.go:122] "RemoveContainer" containerID="3e5a57f44b785397994ce2cdee03b6b5d12d3c9d5a35dc29c5aea62196288801" Oct 30 09:45:05.089434 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:05.089404 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6_odh-ai-gateway-infra(5de510d4-441f-4aa1-98d7-c60848cd26ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" podUID="5de510d4-441f-4aa1-98d7-c60848cd26ed" Oct 30 09:45:05.089541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.089461 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" event={"ID":"5de510d4-441f-4aa1-98d7-c60848cd26ed","Type":"ContainerDied","Data":"3e5a57f44b785397994ce2cdee03b6b5d12d3c9d5a35dc29c5aea62196288801"} Oct 30 09:45:05.091705 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.091679 5938 generic.go:356] "Generic (PLEG): container finished" podID="afc6a7ef-c174-4816-938d-817e57691e6b" containerID="fd62ffce0490dce7be36d758653f245238a8f909f79c3d17482bfc1f582a02d0" exitCode=60 Oct 30 09:45:05.091837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.091767 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" event={"ID":"afc6a7ef-c174-4816-938d-817e57691e6b","Type":"ContainerDied","Data":"fd62ffce0490dce7be36d758653f245238a8f909f79c3d17482bfc1f582a02d0"} Oct 30 09:45:05.092134 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.092111 5938 scope.go:122] "RemoveContainer" containerID="fd62ffce0490dce7be36d758653f245238a8f909f79c3d17482bfc1f582a02d0" Oct 30 09:45:05.092450 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:05.092418 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm_odh-ai-gateway-infra(afc6a7ef-c174-4816-938d-817e57691e6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" podUID="afc6a7ef-c174-4816-938d-817e57691e6b" Oct 30 09:45:05.094876 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.094853 5938 generic.go:356] "Generic (PLEG): container finished" podID="f30e90f0-9862-48cc-8171-df2f58695096" containerID="ea06a40bf02e363db203aef2556942f3efedc74fdd574dc7758ea3ffeefc3567" exitCode=60 Oct 30 09:45:05.094981 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.094955 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" event={"ID":"f30e90f0-9862-48cc-8171-df2f58695096","Type":"ContainerDied","Data":"ea06a40bf02e363db203aef2556942f3efedc74fdd574dc7758ea3ffeefc3567"} Oct 30 09:45:05.095182 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.095162 5938 scope.go:122] "RemoveContainer" containerID="ea06a40bf02e363db203aef2556942f3efedc74fdd574dc7758ea3ffeefc3567" Oct 30 09:45:05.095497 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:05.095474 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26_odh-ai-gateway-infra(f30e90f0-9862-48cc-8171-df2f58695096)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" podUID="f30e90f0-9862-48cc-8171-df2f58695096" Oct 30 09:45:05.097538 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.097515 5938 generic.go:356] "Generic (PLEG): container finished" podID="7b1877d3-aaac-4287-8450-d02cb2497d3f" containerID="8353539443f44a6918c8aedf958602f16accb42bd7cd7e090a48eea90f0bd793" exitCode=60 Oct 30 09:45:05.097676 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.097626 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" event={"ID":"7b1877d3-aaac-4287-8450-d02cb2497d3f","Type":"ContainerDied","Data":"8353539443f44a6918c8aedf958602f16accb42bd7cd7e090a48eea90f0bd793"} Oct 30 09:45:05.097962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.097938 5938 scope.go:122] "RemoveContainer" containerID="8353539443f44a6918c8aedf958602f16accb42bd7cd7e090a48eea90f0bd793" Oct 30 09:45:05.098279 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:05.098255 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8_odh-ai-gateway-infra(7b1877d3-aaac-4287-8450-d02cb2497d3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" podUID="7b1877d3-aaac-4287-8450-d02cb2497d3f" Oct 30 09:45:05.100844 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.100813 5938 generic.go:356] "Generic (PLEG): container finished" podID="ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" containerID="5d57dc0345124fa1885d90fbebf30d711a4ac42281200077f2bbe6467a6df0b7" exitCode=60 Oct 30 09:45:05.101007 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.100991 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" event={"ID":"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4","Type":"ContainerDied","Data":"5d57dc0345124fa1885d90fbebf30d711a4ac42281200077f2bbe6467a6df0b7"} Oct 30 09:45:05.101238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.101221 5938 scope.go:122] "RemoveContainer" containerID="5d57dc0345124fa1885d90fbebf30d711a4ac42281200077f2bbe6467a6df0b7" Oct 30 09:45:05.101494 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:05.101467 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj_odh-ai-gateway-infra(ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" podUID="ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" Oct 30 09:45:05.103460 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.103435 5938 generic.go:356] "Generic (PLEG): container finished" podID="cd723952-1a2b-46d6-96b8-e19056165e36" containerID="4bbc9edbe7c7d30d65aed6a3bf3cb942706cedcc0f0230b1fac90a8bbd1d4c9f" exitCode=60 Oct 30 09:45:05.103567 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.103550 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" event={"ID":"cd723952-1a2b-46d6-96b8-e19056165e36","Type":"ContainerDied","Data":"4bbc9edbe7c7d30d65aed6a3bf3cb942706cedcc0f0230b1fac90a8bbd1d4c9f"} Oct 30 09:45:05.103845 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.103825 5938 scope.go:122] "RemoveContainer" containerID="4bbc9edbe7c7d30d65aed6a3bf3cb942706cedcc0f0230b1fac90a8bbd1d4c9f" Oct 30 09:45:05.104100 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:05.104072 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn_odh-ai-gateway-infra(cd723952-1a2b-46d6-96b8-e19056165e36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" podUID="cd723952-1a2b-46d6-96b8-e19056165e36" Oct 30 09:45:05.106279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.106248 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" event={"ID":"40975fdb-ec84-40e3-8b8f-68bfd44c8123","Type":"ContainerStarted","Data":"6cb7aa7ad4843f0e0bc4b4e9c402136b4895c1005c2fa48dc1475fa5ccc7ee06"} Oct 30 09:45:05.108327 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.108309 5938 generic.go:356] "Generic (PLEG): container finished" podID="ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" containerID="502c2cd7fdd2fa07623450fde9339a64ee9eada850242361b01a7606d2a3aa7c" exitCode=60 Oct 30 09:45:05.108416 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.108349 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" event={"ID":"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4","Type":"ContainerDied","Data":"502c2cd7fdd2fa07623450fde9339a64ee9eada850242361b01a7606d2a3aa7c"} Oct 30 09:45:05.108565 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.108543 5938 scope.go:122] "RemoveContainer" containerID="502c2cd7fdd2fa07623450fde9339a64ee9eada850242361b01a7606d2a3aa7c" Oct 30 09:45:05.108790 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:05.108767 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd_odh-ai-gateway-infra(ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" podUID="ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" Oct 30 09:45:05.125300 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.125278 5938 scope.go:122] "RemoveContainer" containerID="0b40a6b997e1ca8b35d79eb9ba046b4b42adbd5eb28750a75c4cb3f98ef230c3" Oct 30 09:45:05.177301 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.177276 5938 scope.go:122] "RemoveContainer" containerID="cfa8f4a30c608ac82c6d08d1c1f7026e1ea96234746eab50e3ad4915416c8e7a" Oct 30 09:45:05.192425 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.192401 5938 scope.go:122] "RemoveContainer" containerID="6c57dc3fb4f9019742c9c9373b51226adabc13123b6acc6c2d8e18aa5f8a0db1" Oct 30 09:45:05.218094 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.218070 5938 scope.go:122] "RemoveContainer" containerID="f417a8d2a5ab012b0cf10fbf484accb7f0ed8d56638ff940ed3cd2dd9479af4a" Oct 30 09:45:05.232083 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.232062 5938 scope.go:122] "RemoveContainer" containerID="060cc4a01969a3afe2c09bd43a452ea6441084c24607b6caf9dc91ae74f5b669" Oct 30 09:45:05.246451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.246429 5938 scope.go:122] "RemoveContainer" containerID="65a372ab142ae68913d0ef8540926f98bb71f4558cab3d4ebf065266e75ad2f0" Oct 30 09:45:05.260146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.260129 5938 scope.go:122] "RemoveContainer" containerID="be4ab14602642c5327201bec43baeacbbc2fd64d7fbde91c63fc9ed2ed13e55f" Oct 30 09:45:05.273543 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:05.273526 5938 scope.go:122] "RemoveContainer" containerID="a1f718acbed772241777361975ce0b32965c8a542f116efaf623753433cd4904" Oct 30 09:45:06.119908 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:06.119882 5938 scope.go:122] "RemoveContainer" containerID="fd62ffce0490dce7be36d758653f245238a8f909f79c3d17482bfc1f582a02d0" Oct 30 09:45:06.120368 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:06.120239 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm_odh-ai-gateway-infra(afc6a7ef-c174-4816-938d-817e57691e6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" podUID="afc6a7ef-c174-4816-938d-817e57691e6b" Oct 30 09:45:06.121864 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:06.121836 5938 scope.go:122] "RemoveContainer" containerID="ea06a40bf02e363db203aef2556942f3efedc74fdd574dc7758ea3ffeefc3567" Oct 30 09:45:06.122163 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:06.122140 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26_odh-ai-gateway-infra(f30e90f0-9862-48cc-8171-df2f58695096)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" podUID="f30e90f0-9862-48cc-8171-df2f58695096" Oct 30 09:45:06.123571 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:06.123546 5938 scope.go:122] "RemoveContainer" containerID="8353539443f44a6918c8aedf958602f16accb42bd7cd7e090a48eea90f0bd793" Oct 30 09:45:06.123882 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:06.123862 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8_odh-ai-gateway-infra(7b1877d3-aaac-4287-8450-d02cb2497d3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" podUID="7b1877d3-aaac-4287-8450-d02cb2497d3f" Oct 30 09:45:06.125255 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:06.125236 5938 scope.go:122] "RemoveContainer" containerID="5d57dc0345124fa1885d90fbebf30d711a4ac42281200077f2bbe6467a6df0b7" Oct 30 09:45:06.125521 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:06.125493 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj_odh-ai-gateway-infra(ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" podUID="ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" Oct 30 09:45:06.126981 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:06.126965 5938 scope.go:122] "RemoveContainer" containerID="4bbc9edbe7c7d30d65aed6a3bf3cb942706cedcc0f0230b1fac90a8bbd1d4c9f" Oct 30 09:45:06.127188 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:06.127167 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn_odh-ai-gateway-infra(cd723952-1a2b-46d6-96b8-e19056165e36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" podUID="cd723952-1a2b-46d6-96b8-e19056165e36" Oct 30 09:45:06.128747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:06.128721 5938 generic.go:356] "Generic (PLEG): container finished" podID="40975fdb-ec84-40e3-8b8f-68bfd44c8123" containerID="6cb7aa7ad4843f0e0bc4b4e9c402136b4895c1005c2fa48dc1475fa5ccc7ee06" exitCode=60 Oct 30 09:45:06.128850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:06.128753 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" event={"ID":"40975fdb-ec84-40e3-8b8f-68bfd44c8123","Type":"ContainerDied","Data":"6cb7aa7ad4843f0e0bc4b4e9c402136b4895c1005c2fa48dc1475fa5ccc7ee06"} Oct 30 09:45:06.128850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:06.128837 5938 scope.go:122] "RemoveContainer" containerID="262e118ea5d74bd20eecedd54a2833701efa8e071c6079fe39b2384648d46d39" Oct 30 09:45:06.129057 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:06.129041 5938 scope.go:122] "RemoveContainer" containerID="6cb7aa7ad4843f0e0bc4b4e9c402136b4895c1005c2fa48dc1475fa5ccc7ee06" Oct 30 09:45:06.129333 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:06.129294 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889225-fb6zp_odh-ai-gateway-infra(40975fdb-ec84-40e3-8b8f-68bfd44c8123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" podUID="40975fdb-ec84-40e3-8b8f-68bfd44c8123" Oct 30 09:45:06.131134 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:06.131117 5938 scope.go:122] "RemoveContainer" containerID="502c2cd7fdd2fa07623450fde9339a64ee9eada850242361b01a7606d2a3aa7c" Oct 30 09:45:06.131349 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:06.131330 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd_odh-ai-gateway-infra(ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" podUID="ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" Oct 30 09:45:06.132931 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:06.132916 5938 scope.go:122] "RemoveContainer" containerID="f7f6f46dfbe87df03e689146716ddcbc62058ba0bb793c1527e71b5b60e5099a" Oct 30 09:45:06.133147 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:06.133127 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz_odh-ai-gateway-infra(41e9f62f-6ced-4207-a09b-b820f0ad142f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" podUID="41e9f62f-6ced-4207-a09b-b820f0ad142f" Oct 30 09:45:06.134556 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:06.134536 5938 scope.go:122] "RemoveContainer" containerID="15fb72aa6969e3a33e94a40765e5b3878210a81b04e04e71da4fce5ba4b2caf5" Oct 30 09:45:06.134815 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:06.134791 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm_odh-ai-gateway-infra(fc5fe483-3456-4b2f-b626-a14bb00dac59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" podUID="fc5fe483-3456-4b2f-b626-a14bb00dac59" Oct 30 09:45:06.136324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:06.136307 5938 scope.go:122] "RemoveContainer" containerID="3e5a57f44b785397994ce2cdee03b6b5d12d3c9d5a35dc29c5aea62196288801" Oct 30 09:45:06.136535 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:06.136513 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6_odh-ai-gateway-infra(5de510d4-441f-4aa1-98d7-c60848cd26ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" podUID="5de510d4-441f-4aa1-98d7-c60848cd26ed" Oct 30 09:45:07.182440 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:07.180877 5938 scope.go:122] "RemoveContainer" containerID="6cb7aa7ad4843f0e0bc4b4e9c402136b4895c1005c2fa48dc1475fa5ccc7ee06" Oct 30 09:45:07.182440 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:07.181321 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889225-fb6zp_odh-ai-gateway-infra(40975fdb-ec84-40e3-8b8f-68bfd44c8123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" podUID="40975fdb-ec84-40e3-8b8f-68bfd44c8123" Oct 30 09:45:15.250763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:15.250736 5938 scope.go:122] "RemoveContainer" containerID="502c2cd7fdd2fa07623450fde9339a64ee9eada850242361b01a7606d2a3aa7c" Oct 30 09:45:15.815049 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:15.815004 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:45:16.252661 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:16.251446 5938 scope.go:122] "RemoveContainer" containerID="fd62ffce0490dce7be36d758653f245238a8f909f79c3d17482bfc1f582a02d0" Oct 30 09:45:16.254436 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:16.254411 5938 scope.go:122] "RemoveContainer" containerID="8353539443f44a6918c8aedf958602f16accb42bd7cd7e090a48eea90f0bd793" Oct 30 09:45:16.255032 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:16.255012 5938 scope.go:122] "RemoveContainer" containerID="3e5a57f44b785397994ce2cdee03b6b5d12d3c9d5a35dc29c5aea62196288801" Oct 30 09:45:16.255533 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:16.255514 5938 scope.go:122] "RemoveContainer" containerID="6cb7aa7ad4843f0e0bc4b4e9c402136b4895c1005c2fa48dc1475fa5ccc7ee06" Oct 30 09:45:16.342388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:16.342345 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" event={"ID":"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4","Type":"ContainerStarted","Data":"e64cfd073dd4f451fc5fc9d992ae7e859ab92714f1cb9a0a8097257ce65e70ad"} Oct 30 09:45:17.001624 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.001569 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm"] Oct 30 09:45:17.006388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.006370 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:45:17.025254 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.025227 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm"] Oct 30 09:45:17.025353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.025301 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"routing-overlay\"" Oct 30 09:45:17.034998 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.034373 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"kube-root-ca.crt\"" Oct 30 09:45:17.034998 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.034419 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d60c68\"" Oct 30 09:45:17.034998 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.034443 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"payload-processing-external-model-e2e--2110ea34-dockercfg-d56mf\"" Oct 30 09:45:17.034998 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.034472 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"agc-cred-0a2a5c2d6197\"" Oct 30 09:45:17.034998 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.034515 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"agc-cred-b17ddae8de95\"" Oct 30 09:45:17.034998 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.034532 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"openshift-service-ca.crt\"" Oct 30 09:45:17.123669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.123632 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0ad6b7a-5034-44da-94e4-d9e97192b749-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm\" (UID: \"c0ad6b7a-5034-44da-94e4-d9e97192b749\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:45:17.123827 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.123780 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8stx\" (UniqueName: \"kubernetes.io/projected/c0ad6b7a-5034-44da-94e4-d9e97192b749-kube-api-access-s8stx\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm\" (UID: \"c0ad6b7a-5034-44da-94e4-d9e97192b749\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:45:17.123827 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.123823 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ad6b7a-5034-44da-94e4-d9e97192b749-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm\" (UID: \"c0ad6b7a-5034-44da-94e4-d9e97192b749\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:45:17.124003 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.123983 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0ad6b7a-5034-44da-94e4-d9e97192b749-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm\" (UID: \"c0ad6b7a-5034-44da-94e4-d9e97192b749\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:45:17.225898 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.225803 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0ad6b7a-5034-44da-94e4-d9e97192b749-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm\" (UID: \"c0ad6b7a-5034-44da-94e4-d9e97192b749\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:45:17.226082 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.225919 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8stx\" (UniqueName: \"kubernetes.io/projected/c0ad6b7a-5034-44da-94e4-d9e97192b749-kube-api-access-s8stx\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm\" (UID: \"c0ad6b7a-5034-44da-94e4-d9e97192b749\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:45:17.226082 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.225975 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ad6b7a-5034-44da-94e4-d9e97192b749-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm\" (UID: \"c0ad6b7a-5034-44da-94e4-d9e97192b749\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:45:17.226206 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.226121 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0ad6b7a-5034-44da-94e4-d9e97192b749-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm\" (UID: \"c0ad6b7a-5034-44da-94e4-d9e97192b749\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:45:17.226750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.226726 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0ad6b7a-5034-44da-94e4-d9e97192b749-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm\" (UID: \"c0ad6b7a-5034-44da-94e4-d9e97192b749\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:45:17.226881 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.226786 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ad6b7a-5034-44da-94e4-d9e97192b749-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm\" (UID: \"c0ad6b7a-5034-44da-94e4-d9e97192b749\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:45:17.228510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.228485 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0ad6b7a-5034-44da-94e4-d9e97192b749-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm\" (UID: \"c0ad6b7a-5034-44da-94e4-d9e97192b749\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:45:17.247106 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.247077 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8stx\" (UniqueName: \"kubernetes.io/projected/c0ad6b7a-5034-44da-94e4-d9e97192b749-kube-api-access-s8stx\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm\" (UID: \"c0ad6b7a-5034-44da-94e4-d9e97192b749\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:45:17.250455 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.250435 5938 scope.go:122] "RemoveContainer" containerID="5d57dc0345124fa1885d90fbebf30d711a4ac42281200077f2bbe6467a6df0b7" Oct 30 09:45:17.250654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.250636 5938 scope.go:122] "RemoveContainer" containerID="15fb72aa6969e3a33e94a40765e5b3878210a81b04e04e71da4fce5ba4b2caf5" Oct 30 09:45:17.250738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.250724 5938 scope.go:122] "RemoveContainer" containerID="ea06a40bf02e363db203aef2556942f3efedc74fdd574dc7758ea3ffeefc3567" Oct 30 09:45:17.250897 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.250877 5938 scope.go:122] "RemoveContainer" containerID="f7f6f46dfbe87df03e689146716ddcbc62058ba0bb793c1527e71b5b60e5099a" Oct 30 09:45:17.321857 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.321830 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:45:17.357165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.357135 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" event={"ID":"afc6a7ef-c174-4816-938d-817e57691e6b","Type":"ContainerStarted","Data":"48fd04445497b2a5d3faa7a590baa9bac49d3d3de9b169358868191c57e70e8a"} Oct 30 09:45:17.360256 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.360233 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" event={"ID":"7b1877d3-aaac-4287-8450-d02cb2497d3f","Type":"ContainerStarted","Data":"b1a1d03649e86dedf28b22ea2fb8205b52c4028778d954dea099bdf4936bfc20"} Oct 30 09:45:17.362358 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.362335 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" event={"ID":"40975fdb-ec84-40e3-8b8f-68bfd44c8123","Type":"ContainerStarted","Data":"ff9a852c7c948d01733e8bf86e90b326579d81b0a152c86611bc20e78059f2bb"} Oct 30 09:45:17.364295 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.364272 5938 generic.go:356] "Generic (PLEG): container finished" podID="ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" containerID="e64cfd073dd4f451fc5fc9d992ae7e859ab92714f1cb9a0a8097257ce65e70ad" exitCode=60 Oct 30 09:45:17.364366 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.364338 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" event={"ID":"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4","Type":"ContainerDied","Data":"e64cfd073dd4f451fc5fc9d992ae7e859ab92714f1cb9a0a8097257ce65e70ad"} Oct 30 09:45:17.364403 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.364391 5938 scope.go:122] "RemoveContainer" containerID="502c2cd7fdd2fa07623450fde9339a64ee9eada850242361b01a7606d2a3aa7c" Oct 30 09:45:17.364794 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.364779 5938 scope.go:122] "RemoveContainer" containerID="e64cfd073dd4f451fc5fc9d992ae7e859ab92714f1cb9a0a8097257ce65e70ad" Oct 30 09:45:17.365121 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:17.365102 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd_odh-ai-gateway-infra(ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" podUID="ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" Oct 30 09:45:17.367853 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.367828 5938 generic.go:356] "Generic (PLEG): container finished" podID="5de510d4-441f-4aa1-98d7-c60848cd26ed" containerID="96cda40db4e5e46c574e4cbdf7e48a9f68954303f1cb29970e944b00cf445c80" exitCode=60 Oct 30 09:45:17.367944 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.367910 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" event={"ID":"5de510d4-441f-4aa1-98d7-c60848cd26ed","Type":"ContainerDied","Data":"96cda40db4e5e46c574e4cbdf7e48a9f68954303f1cb29970e944b00cf445c80"} Oct 30 09:45:17.368332 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.368311 5938 scope.go:122] "RemoveContainer" containerID="96cda40db4e5e46c574e4cbdf7e48a9f68954303f1cb29970e944b00cf445c80" Oct 30 09:45:17.368681 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:17.368659 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6_odh-ai-gateway-infra(5de510d4-441f-4aa1-98d7-c60848cd26ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" podUID="5de510d4-441f-4aa1-98d7-c60848cd26ed" Oct 30 09:45:17.395170 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.395137 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd"] Oct 30 09:45:17.415015 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.411775 5938 scope.go:122] "RemoveContainer" containerID="3e5a57f44b785397994ce2cdee03b6b5d12d3c9d5a35dc29c5aea62196288801" Oct 30 09:45:17.646549 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:17.646519 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm"] Oct 30 09:45:17.683552 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:45:17.683521 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ad6b7a_5034_44da_94e4_d9e97192b749.slice/crio-8b36c1ef0578d3cf16c6a73cae8b20353f1f986e4e16073f4e79114e0cd7c760 WatchSource:0}: Error finding container 8b36c1ef0578d3cf16c6a73cae8b20353f1f986e4e16073f4e79114e0cd7c760: Status 404 returned error can't find the container with id 8b36c1ef0578d3cf16c6a73cae8b20353f1f986e4e16073f4e79114e0cd7c760 Oct 30 09:45:18.250381 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.250351 5938 scope.go:122] "RemoveContainer" containerID="4bbc9edbe7c7d30d65aed6a3bf3cb942706cedcc0f0230b1fac90a8bbd1d4c9f" Oct 30 09:45:18.378236 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.378198 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" event={"ID":"c0ad6b7a-5034-44da-94e4-d9e97192b749","Type":"ContainerStarted","Data":"e844ddf279d34884ae37d7e3da5fca25084c590c4d88008a708ec928bce929fe"} Oct 30 09:45:18.378699 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.378266 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" event={"ID":"c0ad6b7a-5034-44da-94e4-d9e97192b749","Type":"ContainerStarted","Data":"8b36c1ef0578d3cf16c6a73cae8b20353f1f986e4e16073f4e79114e0cd7c760"} Oct 30 09:45:18.378699 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.378299 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:45:18.380191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.380165 5938 generic.go:356] "Generic (PLEG): container finished" podID="afc6a7ef-c174-4816-938d-817e57691e6b" containerID="48fd04445497b2a5d3faa7a590baa9bac49d3d3de9b169358868191c57e70e8a" exitCode=60 Oct 30 09:45:18.380331 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.380241 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" event={"ID":"afc6a7ef-c174-4816-938d-817e57691e6b","Type":"ContainerDied","Data":"48fd04445497b2a5d3faa7a590baa9bac49d3d3de9b169358868191c57e70e8a"} Oct 30 09:45:18.380331 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.380302 5938 scope.go:122] "RemoveContainer" containerID="fd62ffce0490dce7be36d758653f245238a8f909f79c3d17482bfc1f582a02d0" Oct 30 09:45:18.380633 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.380619 5938 scope.go:122] "RemoveContainer" containerID="48fd04445497b2a5d3faa7a590baa9bac49d3d3de9b169358868191c57e70e8a" Oct 30 09:45:18.380950 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:18.380923 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm_odh-ai-gateway-infra(afc6a7ef-c174-4816-938d-817e57691e6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" podUID="afc6a7ef-c174-4816-938d-817e57691e6b" Oct 30 09:45:18.382465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.382439 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" event={"ID":"f30e90f0-9862-48cc-8171-df2f58695096","Type":"ContainerStarted","Data":"691f9f1973da3839aef64bcdef9e976524d5f8e3f523fc93583d5fc4be952418"} Oct 30 09:45:18.384383 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.384362 5938 generic.go:356] "Generic (PLEG): container finished" podID="7b1877d3-aaac-4287-8450-d02cb2497d3f" containerID="b1a1d03649e86dedf28b22ea2fb8205b52c4028778d954dea099bdf4936bfc20" exitCode=60 Oct 30 09:45:18.384475 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.384427 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" event={"ID":"7b1877d3-aaac-4287-8450-d02cb2497d3f","Type":"ContainerDied","Data":"b1a1d03649e86dedf28b22ea2fb8205b52c4028778d954dea099bdf4936bfc20"} Oct 30 09:45:18.384770 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.384752 5938 scope.go:122] "RemoveContainer" containerID="b1a1d03649e86dedf28b22ea2fb8205b52c4028778d954dea099bdf4936bfc20" Oct 30 09:45:18.385038 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:18.385016 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8_odh-ai-gateway-infra(7b1877d3-aaac-4287-8450-d02cb2497d3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" podUID="7b1877d3-aaac-4287-8450-d02cb2497d3f" Oct 30 09:45:18.386520 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.386499 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" event={"ID":"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4","Type":"ContainerStarted","Data":"569df5c38278fb81bf4f13c6304d60bfc21c149f8f5e2e1f3ebaf579f588aeef"} Oct 30 09:45:18.388456 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.388433 5938 generic.go:356] "Generic (PLEG): container finished" podID="40975fdb-ec84-40e3-8b8f-68bfd44c8123" containerID="ff9a852c7c948d01733e8bf86e90b326579d81b0a152c86611bc20e78059f2bb" exitCode=60 Oct 30 09:45:18.388544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.388497 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" event={"ID":"40975fdb-ec84-40e3-8b8f-68bfd44c8123","Type":"ContainerDied","Data":"ff9a852c7c948d01733e8bf86e90b326579d81b0a152c86611bc20e78059f2bb"} Oct 30 09:45:18.388817 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.388803 5938 scope.go:122] "RemoveContainer" containerID="ff9a852c7c948d01733e8bf86e90b326579d81b0a152c86611bc20e78059f2bb" Oct 30 09:45:18.389058 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:18.389037 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889225-fb6zp_odh-ai-gateway-infra(40975fdb-ec84-40e3-8b8f-68bfd44c8123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" podUID="40975fdb-ec84-40e3-8b8f-68bfd44c8123" Oct 30 09:45:18.392193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.392172 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" event={"ID":"41e9f62f-6ced-4207-a09b-b820f0ad142f","Type":"ContainerStarted","Data":"b3c6d94f17e4e71e020d9a15c6fe88c230d14af24b8424db7e93d2cca19b52ff"} Oct 30 09:45:18.394111 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.394092 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" event={"ID":"fc5fe483-3456-4b2f-b626-a14bb00dac59","Type":"ContainerStarted","Data":"657cbb13af4a64275fbed7e48ecf20f18f58ff350a78ff003e89fbf9286936b7"} Oct 30 09:45:18.412926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.412906 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm"] Oct 30 09:45:18.436739 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.426905 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" podStartSLOduration=2.426888886 podStartE2EDuration="2.426888886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:18.421675453 +0000 UTC m=+3036.284381276" watchObservedRunningTime="2026-10-30 09:45:18.426888886 +0000 UTC m=+3036.289594707" Oct 30 09:45:18.461058 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.461033 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8"] Oct 30 09:45:18.510008 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.509971 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp"] Oct 30 09:45:18.585836 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.585813 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6"] Oct 30 09:45:18.694832 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.693342 5938 scope.go:122] "RemoveContainer" containerID="8353539443f44a6918c8aedf958602f16accb42bd7cd7e090a48eea90f0bd793" Oct 30 09:45:18.724648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.724621 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" Oct 30 09:45:18.725088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.725062 5938 scope.go:122] "RemoveContainer" containerID="6cb7aa7ad4843f0e0bc4b4e9c402136b4895c1005c2fa48dc1475fa5ccc7ee06" Oct 30 09:45:18.750073 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.750053 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" Oct 30 09:45:18.856018 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.855988 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4-kube-api-access-n2qmf\" (UniqueName: \"kubernetes.io/projected/ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4-kube-api-access-n2qmf\") pod \"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4\" (UID: \"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4\") " Oct 30 09:45:18.856350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.856103 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de510d4-441f-4aa1-98d7-c60848cd26ed-kube-api-access-kls2c\" (UniqueName: \"kubernetes.io/projected/5de510d4-441f-4aa1-98d7-c60848cd26ed-kube-api-access-kls2c\") pod \"5de510d4-441f-4aa1-98d7-c60848cd26ed\" (UID: \"5de510d4-441f-4aa1-98d7-c60848cd26ed\") " Oct 30 09:45:18.856458 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.856377 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4-maas-api-service-ca\") pod \"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4\" (UID: \"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4\") " Oct 30 09:45:18.856525 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.856471 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de510d4-441f-4aa1-98d7-c60848cd26ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de510d4-441f-4aa1-98d7-c60848cd26ed-maas-api-service-ca\") pod \"5de510d4-441f-4aa1-98d7-c60848cd26ed\" (UID: \"5de510d4-441f-4aa1-98d7-c60848cd26ed\") " Oct 30 09:45:18.857882 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.857855 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4-maas-api-service-ca" pod "ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" (UID: "ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:18.858003 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.857953 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de510d4-441f-4aa1-98d7-c60848cd26ed-maas-api-service-ca" pod "5de510d4-441f-4aa1-98d7-c60848cd26ed" (UID: "5de510d4-441f-4aa1-98d7-c60848cd26ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:18.858177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.858149 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:18.858177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.858169 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de510d4-441f-4aa1-98d7-c60848cd26ed-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:18.859244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.859224 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de510d4-441f-4aa1-98d7-c60848cd26ed-kube-api-access-kls2c" pod "5de510d4-441f-4aa1-98d7-c60848cd26ed" (UID: "5de510d4-441f-4aa1-98d7-c60848cd26ed"). InnerVolumeSpecName "kube-api-access-kls2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:18.859327 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.859286 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4-kube-api-access-n2qmf" pod "ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" (UID: "ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4"). InnerVolumeSpecName "kube-api-access-n2qmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:18.960060 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.960024 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2qmf\" (UniqueName: \"kubernetes.io/projected/ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4-kube-api-access-n2qmf\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:18.960060 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:18.960053 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kls2c\" (UniqueName: \"kubernetes.io/projected/5de510d4-441f-4aa1-98d7-c60848cd26ed-kube-api-access-kls2c\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:19.409645 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.409619 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" Oct 30 09:45:19.410014 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.409623 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd" event={"ID":"ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4","Type":"ContainerDied","Data":"3ea03e7521dee0d47bfa668f4ff3171f63fa12836b9a903ac51403079e793cdb"} Oct 30 09:45:19.410014 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.409790 5938 scope.go:122] "RemoveContainer" containerID="e64cfd073dd4f451fc5fc9d992ae7e859ab92714f1cb9a0a8097257ce65e70ad" Oct 30 09:45:19.412104 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.412079 5938 generic.go:356] "Generic (PLEG): container finished" podID="41e9f62f-6ced-4207-a09b-b820f0ad142f" containerID="b3c6d94f17e4e71e020d9a15c6fe88c230d14af24b8424db7e93d2cca19b52ff" exitCode=60 Oct 30 09:45:19.412225 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.412120 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" event={"ID":"41e9f62f-6ced-4207-a09b-b820f0ad142f","Type":"ContainerDied","Data":"b3c6d94f17e4e71e020d9a15c6fe88c230d14af24b8424db7e93d2cca19b52ff"} Oct 30 09:45:19.412509 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.412491 5938 scope.go:122] "RemoveContainer" containerID="b3c6d94f17e4e71e020d9a15c6fe88c230d14af24b8424db7e93d2cca19b52ff" Oct 30 09:45:19.412871 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:19.412850 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz_odh-ai-gateway-infra(41e9f62f-6ced-4207-a09b-b820f0ad142f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" podUID="41e9f62f-6ced-4207-a09b-b820f0ad142f" Oct 30 09:45:19.414721 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.414633 5938 generic.go:356] "Generic (PLEG): container finished" podID="fc5fe483-3456-4b2f-b626-a14bb00dac59" containerID="657cbb13af4a64275fbed7e48ecf20f18f58ff350a78ff003e89fbf9286936b7" exitCode=60 Oct 30 09:45:19.414721 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.414661 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" event={"ID":"fc5fe483-3456-4b2f-b626-a14bb00dac59","Type":"ContainerDied","Data":"657cbb13af4a64275fbed7e48ecf20f18f58ff350a78ff003e89fbf9286936b7"} Oct 30 09:45:19.415057 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.415040 5938 scope.go:122] "RemoveContainer" containerID="657cbb13af4a64275fbed7e48ecf20f18f58ff350a78ff003e89fbf9286936b7" Oct 30 09:45:19.415402 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:19.415374 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm_odh-ai-gateway-infra(fc5fe483-3456-4b2f-b626-a14bb00dac59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" podUID="fc5fe483-3456-4b2f-b626-a14bb00dac59" Oct 30 09:45:19.416262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.416245 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" Oct 30 09:45:19.416339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.416281 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6" event={"ID":"5de510d4-441f-4aa1-98d7-c60848cd26ed","Type":"ContainerDied","Data":"4eaeef0f9af36561942213cda6120ccaaa99764b766d24e867341fcb77c60602"} Oct 30 09:45:19.422342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.422320 5938 generic.go:356] "Generic (PLEG): container finished" podID="f30e90f0-9862-48cc-8171-df2f58695096" containerID="691f9f1973da3839aef64bcdef9e976524d5f8e3f523fc93583d5fc4be952418" exitCode=60 Oct 30 09:45:19.422417 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.422406 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" event={"ID":"f30e90f0-9862-48cc-8171-df2f58695096","Type":"ContainerDied","Data":"691f9f1973da3839aef64bcdef9e976524d5f8e3f523fc93583d5fc4be952418"} Oct 30 09:45:19.422916 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.422896 5938 scope.go:122] "RemoveContainer" containerID="691f9f1973da3839aef64bcdef9e976524d5f8e3f523fc93583d5fc4be952418" Oct 30 09:45:19.423209 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:19.423186 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26_odh-ai-gateway-infra(f30e90f0-9862-48cc-8171-df2f58695096)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" podUID="f30e90f0-9862-48cc-8171-df2f58695096" Oct 30 09:45:19.425507 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.425455 5938 scope.go:122] "RemoveContainer" containerID="f7f6f46dfbe87df03e689146716ddcbc62058ba0bb793c1527e71b5b60e5099a" Oct 30 09:45:19.429158 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.429071 5938 generic.go:356] "Generic (PLEG): container finished" podID="ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" containerID="569df5c38278fb81bf4f13c6304d60bfc21c149f8f5e2e1f3ebaf579f588aeef" exitCode=60 Oct 30 09:45:19.429158 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.429131 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" event={"ID":"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4","Type":"ContainerDied","Data":"569df5c38278fb81bf4f13c6304d60bfc21c149f8f5e2e1f3ebaf579f588aeef"} Oct 30 09:45:19.429522 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.429506 5938 scope.go:122] "RemoveContainer" containerID="569df5c38278fb81bf4f13c6304d60bfc21c149f8f5e2e1f3ebaf579f588aeef" Oct 30 09:45:19.429873 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:19.429837 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj_odh-ai-gateway-infra(ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" podUID="ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" Oct 30 09:45:19.434558 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.431881 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" event={"ID":"cd723952-1a2b-46d6-96b8-e19056165e36","Type":"ContainerStarted","Data":"c2ec2efe3d21bf1d2d4dcb1afd89d5918298c272f06d68121036b42c2e75adf6"} Oct 30 09:45:19.505799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.493661 5938 scope.go:122] "RemoveContainer" containerID="15fb72aa6969e3a33e94a40765e5b3878210a81b04e04e71da4fce5ba4b2caf5" Oct 30 09:45:19.533943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.533917 5938 scope.go:122] "RemoveContainer" containerID="96cda40db4e5e46c574e4cbdf7e48a9f68954303f1cb29970e944b00cf445c80" Oct 30 09:45:19.564950 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.564675 5938 scope.go:122] "RemoveContainer" containerID="ea06a40bf02e363db203aef2556942f3efedc74fdd574dc7758ea3ffeefc3567" Oct 30 09:45:19.584002 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.583652 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj"] Oct 30 09:45:19.632540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.632279 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz"] Oct 30 09:45:19.633090 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.633063 5938 scope.go:122] "RemoveContainer" containerID="5d57dc0345124fa1885d90fbebf30d711a4ac42281200077f2bbe6467a6df0b7" Oct 30 09:45:19.636435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.636407 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" Oct 30 09:45:19.679881 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.679855 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" Oct 30 09:45:19.685177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.685155 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" Oct 30 09:45:19.715384 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.715361 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm"] Oct 30 09:45:19.761931 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.761908 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26"] Oct 30 09:45:19.781758 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.781724 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b1877d3-aaac-4287-8450-d02cb2497d3f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1877d3-aaac-4287-8450-d02cb2497d3f-maas-api-service-ca\") pod \"7b1877d3-aaac-4287-8450-d02cb2497d3f\" (UID: \"7b1877d3-aaac-4287-8450-d02cb2497d3f\") " Oct 30 09:45:19.781886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.781858 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afc6a7ef-c174-4816-938d-817e57691e6b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc6a7ef-c174-4816-938d-817e57691e6b-maas-api-service-ca\") pod \"afc6a7ef-c174-4816-938d-817e57691e6b\" (UID: \"afc6a7ef-c174-4816-938d-817e57691e6b\") " Oct 30 09:45:19.781970 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.781923 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc6a7ef-c174-4816-938d-817e57691e6b-kube-api-access-dqkz2\" (UniqueName: \"kubernetes.io/projected/afc6a7ef-c174-4816-938d-817e57691e6b-kube-api-access-dqkz2\") pod \"afc6a7ef-c174-4816-938d-817e57691e6b\" (UID: \"afc6a7ef-c174-4816-938d-817e57691e6b\") " Oct 30 09:45:19.782146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.782129 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b1877d3-aaac-4287-8450-d02cb2497d3f-kube-api-access-97btn\" (UniqueName: \"kubernetes.io/projected/7b1877d3-aaac-4287-8450-d02cb2497d3f-kube-api-access-97btn\") pod \"7b1877d3-aaac-4287-8450-d02cb2497d3f\" (UID: \"7b1877d3-aaac-4287-8450-d02cb2497d3f\") " Oct 30 09:45:19.782221 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.782152 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1877d3-aaac-4287-8450-d02cb2497d3f-maas-api-service-ca" pod "7b1877d3-aaac-4287-8450-d02cb2497d3f" (UID: "7b1877d3-aaac-4287-8450-d02cb2497d3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:19.782221 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.782206 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40975fdb-ec84-40e3-8b8f-68bfd44c8123-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40975fdb-ec84-40e3-8b8f-68bfd44c8123-maas-api-service-ca\") pod \"40975fdb-ec84-40e3-8b8f-68bfd44c8123\" (UID: \"40975fdb-ec84-40e3-8b8f-68bfd44c8123\") " Oct 30 09:45:19.782337 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.782216 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc6a7ef-c174-4816-938d-817e57691e6b-maas-api-service-ca" pod "afc6a7ef-c174-4816-938d-817e57691e6b" (UID: "afc6a7ef-c174-4816-938d-817e57691e6b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:19.782337 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.782242 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40975fdb-ec84-40e3-8b8f-68bfd44c8123-kube-api-access-ch896\" (UniqueName: \"kubernetes.io/projected/40975fdb-ec84-40e3-8b8f-68bfd44c8123-kube-api-access-ch896\") pod \"40975fdb-ec84-40e3-8b8f-68bfd44c8123\" (UID: \"40975fdb-ec84-40e3-8b8f-68bfd44c8123\") " Oct 30 09:45:19.782691 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.782667 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40975fdb-ec84-40e3-8b8f-68bfd44c8123-maas-api-service-ca" pod "40975fdb-ec84-40e3-8b8f-68bfd44c8123" (UID: "40975fdb-ec84-40e3-8b8f-68bfd44c8123"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:19.783193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.783174 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40975fdb-ec84-40e3-8b8f-68bfd44c8123-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:19.783248 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.783201 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1877d3-aaac-4287-8450-d02cb2497d3f-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:19.783248 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.783216 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc6a7ef-c174-4816-938d-817e57691e6b-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:19.784432 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.784406 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc6a7ef-c174-4816-938d-817e57691e6b-kube-api-access-dqkz2" pod "afc6a7ef-c174-4816-938d-817e57691e6b" (UID: "afc6a7ef-c174-4816-938d-817e57691e6b"). InnerVolumeSpecName "kube-api-access-dqkz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:19.784622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.784572 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40975fdb-ec84-40e3-8b8f-68bfd44c8123-kube-api-access-ch896" pod "40975fdb-ec84-40e3-8b8f-68bfd44c8123" (UID: "40975fdb-ec84-40e3-8b8f-68bfd44c8123"). InnerVolumeSpecName "kube-api-access-ch896". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:19.784686 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.784616 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1877d3-aaac-4287-8450-d02cb2497d3f-kube-api-access-97btn" pod "7b1877d3-aaac-4287-8450-d02cb2497d3f" (UID: "7b1877d3-aaac-4287-8450-d02cb2497d3f"). InnerVolumeSpecName "kube-api-access-97btn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:19.884896 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.884817 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97btn\" (UniqueName: \"kubernetes.io/projected/7b1877d3-aaac-4287-8450-d02cb2497d3f-kube-api-access-97btn\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:19.884896 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.884852 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch896\" (UniqueName: \"kubernetes.io/projected/40975fdb-ec84-40e3-8b8f-68bfd44c8123-kube-api-access-ch896\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:19.884896 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.884867 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqkz2\" (UniqueName: \"kubernetes.io/projected/afc6a7ef-c174-4816-938d-817e57691e6b-kube-api-access-dqkz2\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:19.917623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.917557 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6"] Oct 30 09:45:19.936505 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:19.936479 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fc7ee7-29889225-x5mf6"] Oct 30 09:45:20.136468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.136374 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd"] Oct 30 09:45:20.156073 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.156034 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-176b6c-29889225-65wvd"] Oct 30 09:45:20.264077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.264041 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de510d4-441f-4aa1-98d7-c60848cd26ed" path="/var/lib/kubelet/pods/5de510d4-441f-4aa1-98d7-c60848cd26ed/volumes" Oct 30 09:45:20.264571 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.264553 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" path="/var/lib/kubelet/pods/ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4/volumes" Oct 30 09:45:20.440820 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.440741 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" Oct 30 09:45:20.440820 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.440767 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp" event={"ID":"40975fdb-ec84-40e3-8b8f-68bfd44c8123","Type":"ContainerDied","Data":"2b71a9fa5dd04a76434a370c490e9c3698cc2a0383c75ab93ba71119a7bf5994"} Oct 30 09:45:20.441319 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.440854 5938 scope.go:122] "RemoveContainer" containerID="ff9a852c7c948d01733e8bf86e90b326579d81b0a152c86611bc20e78059f2bb" Oct 30 09:45:20.450254 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.450229 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" event={"ID":"afc6a7ef-c174-4816-938d-817e57691e6b","Type":"ContainerDied","Data":"bbec99ecb0fd26ef8e3439f17e6c0d3cc9f165bddc703a7d63463f327a0476e3"} Oct 30 09:45:20.450425 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.450240 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm" Oct 30 09:45:20.454330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.454296 5938 scope.go:122] "RemoveContainer" containerID="48fd04445497b2a5d3faa7a590baa9bac49d3d3de9b169358868191c57e70e8a" Oct 30 09:45:20.456286 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.456261 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" event={"ID":"7b1877d3-aaac-4287-8450-d02cb2497d3f","Type":"ContainerDied","Data":"671c616185f9fd2c35d2b5234b740554af2ebeecc5a3f55dedc5c4892513ca21"} Oct 30 09:45:20.456286 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.456282 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8" Oct 30 09:45:20.462603 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.462560 5938 generic.go:356] "Generic (PLEG): container finished" podID="cd723952-1a2b-46d6-96b8-e19056165e36" containerID="c2ec2efe3d21bf1d2d4dcb1afd89d5918298c272f06d68121036b42c2e75adf6" exitCode=60 Oct 30 09:45:20.462712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.462646 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" event={"ID":"cd723952-1a2b-46d6-96b8-e19056165e36","Type":"ContainerDied","Data":"c2ec2efe3d21bf1d2d4dcb1afd89d5918298c272f06d68121036b42c2e75adf6"} Oct 30 09:45:20.463133 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.463110 5938 scope.go:122] "RemoveContainer" containerID="c2ec2efe3d21bf1d2d4dcb1afd89d5918298c272f06d68121036b42c2e75adf6" Oct 30 09:45:20.463484 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:20.463461 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn_odh-ai-gateway-infra(cd723952-1a2b-46d6-96b8-e19056165e36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" podUID="cd723952-1a2b-46d6-96b8-e19056165e36" Oct 30 09:45:20.468509 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.468483 5938 scope.go:122] "RemoveContainer" containerID="b1a1d03649e86dedf28b22ea2fb8205b52c4028778d954dea099bdf4936bfc20" Oct 30 09:45:20.488138 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.488111 5938 scope.go:122] "RemoveContainer" containerID="4bbc9edbe7c7d30d65aed6a3bf3cb942706cedcc0f0230b1fac90a8bbd1d4c9f" Oct 30 09:45:20.546765 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.546695 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm"] Oct 30 09:45:20.552453 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.552422 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa2fdf-29889225-k6pwm"] Oct 30 09:45:20.600425 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.600394 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp"] Oct 30 09:45:20.604331 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.604289 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889225-fb6zp"] Oct 30 09:45:20.696543 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.696448 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" Oct 30 09:45:20.706983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.706960 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" Oct 30 09:45:20.722986 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.722956 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" Oct 30 09:45:20.742107 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.742082 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" Oct 30 09:45:20.775612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.775569 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8"] Oct 30 09:45:20.786180 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.786152 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e3d2ff-29889225-tlzb8"] Oct 30 09:45:20.789832 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.789808 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn"] Oct 30 09:45:20.799300 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.799273 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e9f62f-6ced-4207-a09b-b820f0ad142f-kube-api-access-c6ll8\" (UniqueName: \"kubernetes.io/projected/41e9f62f-6ced-4207-a09b-b820f0ad142f-kube-api-access-c6ll8\") pod \"41e9f62f-6ced-4207-a09b-b820f0ad142f\" (UID: \"41e9f62f-6ced-4207-a09b-b820f0ad142f\") " Oct 30 09:45:20.799449 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.799414 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e9f62f-6ced-4207-a09b-b820f0ad142f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e9f62f-6ced-4207-a09b-b820f0ad142f-maas-api-service-ca\") pod \"41e9f62f-6ced-4207-a09b-b820f0ad142f\" (UID: \"41e9f62f-6ced-4207-a09b-b820f0ad142f\") " Oct 30 09:45:20.799804 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.799783 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e9f62f-6ced-4207-a09b-b820f0ad142f-maas-api-service-ca" pod "41e9f62f-6ced-4207-a09b-b820f0ad142f" (UID: "41e9f62f-6ced-4207-a09b-b820f0ad142f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:20.799942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.799929 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e9f62f-6ced-4207-a09b-b820f0ad142f-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:20.802154 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.802131 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e9f62f-6ced-4207-a09b-b820f0ad142f-kube-api-access-c6ll8" pod "41e9f62f-6ced-4207-a09b-b820f0ad142f" (UID: "41e9f62f-6ced-4207-a09b-b820f0ad142f"). InnerVolumeSpecName "kube-api-access-c6ll8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:20.901322 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.901294 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5fe483-3456-4b2f-b626-a14bb00dac59-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc5fe483-3456-4b2f-b626-a14bb00dac59-maas-api-service-ca\") pod \"fc5fe483-3456-4b2f-b626-a14bb00dac59\" (UID: \"fc5fe483-3456-4b2f-b626-a14bb00dac59\") " Oct 30 09:45:20.901644 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.901339 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4-kube-api-access-px48k\" (UniqueName: \"kubernetes.io/projected/ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4-kube-api-access-px48k\") pod \"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4\" (UID: \"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4\") " Oct 30 09:45:20.901644 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.901419 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30e90f0-9862-48cc-8171-df2f58695096-kube-api-access-j4gcf\" (UniqueName: \"kubernetes.io/projected/f30e90f0-9862-48cc-8171-df2f58695096-kube-api-access-j4gcf\") pod \"f30e90f0-9862-48cc-8171-df2f58695096\" (UID: \"f30e90f0-9862-48cc-8171-df2f58695096\") " Oct 30 09:45:20.901644 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.901630 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4-maas-api-service-ca\") pod \"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4\" (UID: \"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4\") " Oct 30 09:45:20.901849 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.901695 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5fe483-3456-4b2f-b626-a14bb00dac59-maas-api-service-ca" pod "fc5fe483-3456-4b2f-b626-a14bb00dac59" (UID: "fc5fe483-3456-4b2f-b626-a14bb00dac59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:20.902081 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.902028 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4-maas-api-service-ca" pod "ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" (UID: "ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:20.902215 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.902130 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5fe483-3456-4b2f-b626-a14bb00dac59-kube-api-access-gmhff\" (UniqueName: \"kubernetes.io/projected/fc5fe483-3456-4b2f-b626-a14bb00dac59-kube-api-access-gmhff\") pod \"fc5fe483-3456-4b2f-b626-a14bb00dac59\" (UID: \"fc5fe483-3456-4b2f-b626-a14bb00dac59\") " Oct 30 09:45:20.903067 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.902684 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f30e90f0-9862-48cc-8171-df2f58695096-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f30e90f0-9862-48cc-8171-df2f58695096-maas-api-service-ca\") pod \"f30e90f0-9862-48cc-8171-df2f58695096\" (UID: \"f30e90f0-9862-48cc-8171-df2f58695096\") " Oct 30 09:45:20.903067 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.903033 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30e90f0-9862-48cc-8171-df2f58695096-maas-api-service-ca" pod "f30e90f0-9862-48cc-8171-df2f58695096" (UID: "f30e90f0-9862-48cc-8171-df2f58695096"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:20.903546 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.903518 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f30e90f0-9862-48cc-8171-df2f58695096-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:20.903712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.903552 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc5fe483-3456-4b2f-b626-a14bb00dac59-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:20.903712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.903569 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6ll8\" (UniqueName: \"kubernetes.io/projected/41e9f62f-6ced-4207-a09b-b820f0ad142f-kube-api-access-c6ll8\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:20.903712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.903607 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:20.904389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.904365 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30e90f0-9862-48cc-8171-df2f58695096-kube-api-access-j4gcf" pod "f30e90f0-9862-48cc-8171-df2f58695096" (UID: "f30e90f0-9862-48cc-8171-df2f58695096"). InnerVolumeSpecName "kube-api-access-j4gcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:20.904507 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.904397 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4-kube-api-access-px48k" pod "ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" (UID: "ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4"). InnerVolumeSpecName "kube-api-access-px48k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:20.905901 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:20.905876 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5fe483-3456-4b2f-b626-a14bb00dac59-kube-api-access-gmhff" pod "fc5fe483-3456-4b2f-b626-a14bb00dac59" (UID: "fc5fe483-3456-4b2f-b626-a14bb00dac59"). InnerVolumeSpecName "kube-api-access-gmhff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:21.005238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.005152 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px48k\" (UniqueName: \"kubernetes.io/projected/ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4-kube-api-access-px48k\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:21.005238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.005189 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4gcf\" (UniqueName: \"kubernetes.io/projected/f30e90f0-9862-48cc-8171-df2f58695096-kube-api-access-j4gcf\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:21.005238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.005204 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmhff\" (UniqueName: \"kubernetes.io/projected/fc5fe483-3456-4b2f-b626-a14bb00dac59-kube-api-access-gmhff\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:21.477739 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.477697 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" event={"ID":"f30e90f0-9862-48cc-8171-df2f58695096","Type":"ContainerDied","Data":"2b2ccd5d2a87e1433b3c867f51aca0865e15959253d4496accbdbf7763ebeb5e"} Oct 30 09:45:21.478212 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.477770 5938 scope.go:122] "RemoveContainer" containerID="691f9f1973da3839aef64bcdef9e976524d5f8e3f523fc93583d5fc4be952418" Oct 30 09:45:21.478212 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.477939 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26" Oct 30 09:45:21.480503 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.480475 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" Oct 30 09:45:21.480660 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.480513 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj" event={"ID":"ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4","Type":"ContainerDied","Data":"d1f2b85ba1714210d54bc1292271768ab142b3438b67926c491ac72c71b86936"} Oct 30 09:45:21.485922 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.485895 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" event={"ID":"41e9f62f-6ced-4207-a09b-b820f0ad142f","Type":"ContainerDied","Data":"8eccb25ac8dcc39270ca37e42e5611f64aedbae9af0d635801a30e2bd1d618dc"} Oct 30 09:45:21.486046 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.485932 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz" Oct 30 09:45:21.491863 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.491842 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" Oct 30 09:45:21.492002 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.491955 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm" event={"ID":"fc5fe483-3456-4b2f-b626-a14bb00dac59","Type":"ContainerDied","Data":"81defbe9aa9774b76eb7d51d596b52578cb64e3d9e7148a11e3502cd6e5dec30"} Oct 30 09:45:21.493050 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.493033 5938 scope.go:122] "RemoveContainer" containerID="569df5c38278fb81bf4f13c6304d60bfc21c149f8f5e2e1f3ebaf579f588aeef" Oct 30 09:45:21.504902 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.504881 5938 scope.go:122] "RemoveContainer" containerID="b3c6d94f17e4e71e020d9a15c6fe88c230d14af24b8424db7e93d2cca19b52ff" Oct 30 09:45:21.516030 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.516012 5938 scope.go:122] "RemoveContainer" containerID="657cbb13af4a64275fbed7e48ecf20f18f58ff350a78ff003e89fbf9286936b7" Oct 30 09:45:21.610413 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.610384 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26"] Oct 30 09:45:21.614632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.614603 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3d9b36-29889225-j2c26"] Oct 30 09:45:21.661139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.661110 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" Oct 30 09:45:21.663168 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.663149 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj"] Oct 30 09:45:21.671723 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.671702 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f6f96-29889225-nv5pj"] Oct 30 09:45:21.707639 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.707611 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz"] Oct 30 09:45:21.711267 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.711246 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889225-rvhgz"] Oct 30 09:45:21.747130 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.747094 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm"] Oct 30 09:45:21.750182 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.750164 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0dc148-29889225-bbbrm"] Oct 30 09:45:21.815488 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.815456 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd723952-1a2b-46d6-96b8-e19056165e36-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd723952-1a2b-46d6-96b8-e19056165e36-maas-api-service-ca\") pod \"cd723952-1a2b-46d6-96b8-e19056165e36\" (UID: \"cd723952-1a2b-46d6-96b8-e19056165e36\") " Oct 30 09:45:21.815698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.815673 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd723952-1a2b-46d6-96b8-e19056165e36-kube-api-access-2l6nz\" (UniqueName: \"kubernetes.io/projected/cd723952-1a2b-46d6-96b8-e19056165e36-kube-api-access-2l6nz\") pod \"cd723952-1a2b-46d6-96b8-e19056165e36\" (UID: \"cd723952-1a2b-46d6-96b8-e19056165e36\") " Oct 30 09:45:21.815909 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.815886 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd723952-1a2b-46d6-96b8-e19056165e36-maas-api-service-ca" pod "cd723952-1a2b-46d6-96b8-e19056165e36" (UID: "cd723952-1a2b-46d6-96b8-e19056165e36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:21.816367 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.816340 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd723952-1a2b-46d6-96b8-e19056165e36-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:21.817982 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.817961 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd723952-1a2b-46d6-96b8-e19056165e36-kube-api-access-2l6nz" pod "cd723952-1a2b-46d6-96b8-e19056165e36" (UID: "cd723952-1a2b-46d6-96b8-e19056165e36"). InnerVolumeSpecName "kube-api-access-2l6nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:21.917525 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:21.917450 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l6nz\" (UniqueName: \"kubernetes.io/projected/cd723952-1a2b-46d6-96b8-e19056165e36-kube-api-access-2l6nz\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:45:22.271527 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:22.271443 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40975fdb-ec84-40e3-8b8f-68bfd44c8123" path="/var/lib/kubelet/pods/40975fdb-ec84-40e3-8b8f-68bfd44c8123/volumes" Oct 30 09:45:22.271935 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:22.271919 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e9f62f-6ced-4207-a09b-b820f0ad142f" path="/var/lib/kubelet/pods/41e9f62f-6ced-4207-a09b-b820f0ad142f/volumes" Oct 30 09:45:22.272313 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:22.272300 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b1877d3-aaac-4287-8450-d02cb2497d3f" path="/var/lib/kubelet/pods/7b1877d3-aaac-4287-8450-d02cb2497d3f/volumes" Oct 30 09:45:22.273027 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:22.273012 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afc6a7ef-c174-4816-938d-817e57691e6b" path="/var/lib/kubelet/pods/afc6a7ef-c174-4816-938d-817e57691e6b/volumes" Oct 30 09:45:22.273387 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:22.273375 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" path="/var/lib/kubelet/pods/ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4/volumes" Oct 30 09:45:22.273766 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:22.273753 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f30e90f0-9862-48cc-8171-df2f58695096" path="/var/lib/kubelet/pods/f30e90f0-9862-48cc-8171-df2f58695096/volumes" Oct 30 09:45:22.274380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:22.274367 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5fe483-3456-4b2f-b626-a14bb00dac59" path="/var/lib/kubelet/pods/fc5fe483-3456-4b2f-b626-a14bb00dac59/volumes" Oct 30 09:45:22.502821 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:22.502790 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" Oct 30 09:45:22.502821 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:22.502801 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn" event={"ID":"cd723952-1a2b-46d6-96b8-e19056165e36","Type":"ContainerDied","Data":"5e8bde6700cc26f3de152588d82a28c1e4c8d9fa5c79bf1fa9ee3bf31790d70c"} Oct 30 09:45:22.503414 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:22.502859 5938 scope.go:122] "RemoveContainer" containerID="c2ec2efe3d21bf1d2d4dcb1afd89d5918298c272f06d68121036b42c2e75adf6" Oct 30 09:45:22.559301 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:22.559121 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn"] Oct 30 09:45:22.572133 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:22.572109 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889225-75bmn"] Oct 30 09:45:24.259623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:24.259567 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd723952-1a2b-46d6-96b8-e19056165e36" path="/var/lib/kubelet/pods/cd723952-1a2b-46d6-96b8-e19056165e36/volumes" Oct 30 09:45:29.435322 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:29.435297 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:45:43.890601 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:43.890567 5938 scope.go:122] "RemoveContainer" containerID="ebc4f305a4b8128c125b5759ea5c31725a719a25de9d09b0c3c927e744d11cf8" Oct 30 09:45:43.901002 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:43.900981 5938 scope.go:122] "RemoveContainer" containerID="e8069d7e140878ba5fd9617f6774d40aa07dbf68dfd737b0c7560183d583fdb9" Oct 30 09:45:45.809900 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:45:45.809866 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:45:53.844476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.844394 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9"] Oct 30 09:45:53.845382 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845360 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" containerName="cleanup" Oct 30 09:45:53.845499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845383 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" containerName="cleanup" Oct 30 09:45:53.845499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845403 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f30e90f0-9862-48cc-8171-df2f58695096" containerName="cleanup" Oct 30 09:45:53.845499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845409 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30e90f0-9862-48cc-8171-df2f58695096" containerName="cleanup" Oct 30 09:45:53.845499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845424 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" containerName="cleanup" Oct 30 09:45:53.845499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845431 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" containerName="cleanup" Oct 30 09:45:53.845499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845445 5938 cpu_manager.go:414] "containerMap: removing container" podUID="41e9f62f-6ced-4207-a09b-b820f0ad142f" containerName="cleanup" Oct 30 09:45:53.845499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845453 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e9f62f-6ced-4207-a09b-b820f0ad142f" containerName="cleanup" Oct 30 09:45:53.845499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845472 5938 cpu_manager.go:414] "containerMap: removing container" podUID="afc6a7ef-c174-4816-938d-817e57691e6b" containerName="cleanup" Oct 30 09:45:53.845499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845479 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc6a7ef-c174-4816-938d-817e57691e6b" containerName="cleanup" Oct 30 09:45:53.845499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845487 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f30e90f0-9862-48cc-8171-df2f58695096" containerName="cleanup" Oct 30 09:45:53.845499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845493 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30e90f0-9862-48cc-8171-df2f58695096" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845505 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5de510d4-441f-4aa1-98d7-c60848cd26ed" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845512 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de510d4-441f-4aa1-98d7-c60848cd26ed" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845521 5938 cpu_manager.go:414] "containerMap: removing container" podUID="fc5fe483-3456-4b2f-b626-a14bb00dac59" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845528 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5fe483-3456-4b2f-b626-a14bb00dac59" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845542 5938 cpu_manager.go:414] "containerMap: removing container" podUID="afc6a7ef-c174-4816-938d-817e57691e6b" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845549 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc6a7ef-c174-4816-938d-817e57691e6b" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845559 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845566 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845577 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845601 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845618 5938 cpu_manager.go:414] "containerMap: removing container" podUID="41e9f62f-6ced-4207-a09b-b820f0ad142f" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845624 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e9f62f-6ced-4207-a09b-b820f0ad142f" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845633 5938 cpu_manager.go:414] "containerMap: removing container" podUID="fc5fe483-3456-4b2f-b626-a14bb00dac59" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845640 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5fe483-3456-4b2f-b626-a14bb00dac59" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845658 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7b1877d3-aaac-4287-8450-d02cb2497d3f" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845665 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b1877d3-aaac-4287-8450-d02cb2497d3f" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845687 5938 cpu_manager.go:414] "containerMap: removing container" podUID="cd723952-1a2b-46d6-96b8-e19056165e36" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845694 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd723952-1a2b-46d6-96b8-e19056165e36" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845707 5938 cpu_manager.go:414] "containerMap: removing container" podUID="40975fdb-ec84-40e3-8b8f-68bfd44c8123" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845716 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40975fdb-ec84-40e3-8b8f-68bfd44c8123" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845725 5938 cpu_manager.go:414] "containerMap: removing container" podUID="afc6a7ef-c174-4816-938d-817e57691e6b" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845731 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc6a7ef-c174-4816-938d-817e57691e6b" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845753 5938 cpu_manager.go:414] "containerMap: removing container" podUID="cd723952-1a2b-46d6-96b8-e19056165e36" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845760 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd723952-1a2b-46d6-96b8-e19056165e36" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845769 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f30e90f0-9862-48cc-8171-df2f58695096" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845776 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30e90f0-9862-48cc-8171-df2f58695096" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845798 5938 cpu_manager.go:414] "containerMap: removing container" podUID="40975fdb-ec84-40e3-8b8f-68bfd44c8123" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845806 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40975fdb-ec84-40e3-8b8f-68bfd44c8123" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845820 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5de510d4-441f-4aa1-98d7-c60848cd26ed" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845827 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de510d4-441f-4aa1-98d7-c60848cd26ed" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845839 5938 cpu_manager.go:414] "containerMap: removing container" podUID="cd723952-1a2b-46d6-96b8-e19056165e36" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845846 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd723952-1a2b-46d6-96b8-e19056165e36" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845871 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7b1877d3-aaac-4287-8450-d02cb2497d3f" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845877 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b1877d3-aaac-4287-8450-d02cb2497d3f" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845889 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5de510d4-441f-4aa1-98d7-c60848cd26ed" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.845895 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de510d4-441f-4aa1-98d7-c60848cd26ed" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846078 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc6a7ef-c174-4816-938d-817e57691e6b" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846091 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846102 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846112 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e9f62f-6ced-4207-a09b-b820f0ad142f" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846121 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5fe483-3456-4b2f-b626-a14bb00dac59" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846131 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b1877d3-aaac-4287-8450-d02cb2497d3f" containerName="cleanup" Oct 30 09:45:53.846121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846141 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b1877d3-aaac-4287-8450-d02cb2497d3f" containerName="cleanup" Oct 30 09:45:53.848324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846151 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd723952-1a2b-46d6-96b8-e19056165e36" containerName="cleanup" Oct 30 09:45:53.848324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846159 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="40975fdb-ec84-40e3-8b8f-68bfd44c8123" containerName="cleanup" Oct 30 09:45:53.848324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846168 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd723952-1a2b-46d6-96b8-e19056165e36" containerName="cleanup" Oct 30 09:45:53.848324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846177 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30e90f0-9862-48cc-8171-df2f58695096" containerName="cleanup" Oct 30 09:45:53.848324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846185 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5fe483-3456-4b2f-b626-a14bb00dac59" containerName="cleanup" Oct 30 09:45:53.848324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846195 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de510d4-441f-4aa1-98d7-c60848cd26ed" containerName="cleanup" Oct 30 09:45:53.848324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846203 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e9f62f-6ced-4207-a09b-b820f0ad142f" containerName="cleanup" Oct 30 09:45:53.848324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846211 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd723952-1a2b-46d6-96b8-e19056165e36" containerName="cleanup" Oct 30 09:45:53.848324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846225 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b1877d3-aaac-4287-8450-d02cb2497d3f" containerName="cleanup" Oct 30 09:45:53.848324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846234 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="40975fdb-ec84-40e3-8b8f-68bfd44c8123" containerName="cleanup" Oct 30 09:45:53.848324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846244 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" containerName="cleanup" Oct 30 09:45:53.848324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846251 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30e90f0-9862-48cc-8171-df2f58695096" containerName="cleanup" Oct 30 09:45:53.848324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846261 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" containerName="cleanup" Oct 30 09:45:53.848324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846270 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e9f62f-6ced-4207-a09b-b820f0ad142f" containerName="cleanup" Oct 30 09:45:53.848324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846281 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc6a7ef-c174-4816-938d-817e57691e6b" containerName="cleanup" Oct 30 09:45:53.848324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.846290 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de510d4-441f-4aa1-98d7-c60848cd26ed" containerName="cleanup" Oct 30 09:45:53.852631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.852579 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:45:53.866773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.866751 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9"] Oct 30 09:45:53.868113 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.868087 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rj7g\" (UniqueName: \"kubernetes.io/projected/7d0be4a7-7a63-495e-a4df-560de711b86d-kube-api-access-2rj7g\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9\" (UID: \"7d0be4a7-7a63-495e-a4df-560de711b86d\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:45:53.868167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.868146 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d0be4a7-7a63-495e-a4df-560de711b86d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9\" (UID: \"7d0be4a7-7a63-495e-a4df-560de711b86d\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:45:53.868230 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.868214 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0be4a7-7a63-495e-a4df-560de711b86d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9\" (UID: \"7d0be4a7-7a63-495e-a4df-560de711b86d\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:45:53.868269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.868246 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d0be4a7-7a63-495e-a4df-560de711b86d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9\" (UID: \"7d0be4a7-7a63-495e-a4df-560de711b86d\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:45:53.871261 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.871244 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"e2e-cfl-dfc25b-provider-cred\"" Oct 30 09:45:53.871347 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.871289 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"openshift-service-ca.crt\"" Oct 30 09:45:53.875687 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.875669 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"kube-root-ca.crt\"" Oct 30 09:45:53.875844 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.875759 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"payload-processing-external-model-plugins-e2e-extmod-a-88cbc7\"" Oct 30 09:45:53.875922 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.875902 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"routing-overlay\"" Oct 30 09:45:53.885058 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.885037 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"payload-processing-external-model-e2e--185456a2-dockercfg-2j9zz\"" Oct 30 09:45:53.968981 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.968961 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rj7g\" (UniqueName: \"kubernetes.io/projected/7d0be4a7-7a63-495e-a4df-560de711b86d-kube-api-access-2rj7g\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9\" (UID: \"7d0be4a7-7a63-495e-a4df-560de711b86d\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:45:53.969102 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.969000 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d0be4a7-7a63-495e-a4df-560de711b86d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9\" (UID: \"7d0be4a7-7a63-495e-a4df-560de711b86d\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:45:53.969102 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.969027 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0be4a7-7a63-495e-a4df-560de711b86d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9\" (UID: \"7d0be4a7-7a63-495e-a4df-560de711b86d\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:45:53.969102 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.969051 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d0be4a7-7a63-495e-a4df-560de711b86d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9\" (UID: \"7d0be4a7-7a63-495e-a4df-560de711b86d\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:45:53.969803 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.969785 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d0be4a7-7a63-495e-a4df-560de711b86d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9\" (UID: \"7d0be4a7-7a63-495e-a4df-560de711b86d\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:45:53.969878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.969812 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0be4a7-7a63-495e-a4df-560de711b86d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9\" (UID: \"7d0be4a7-7a63-495e-a4df-560de711b86d\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:45:53.971672 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.971647 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d0be4a7-7a63-495e-a4df-560de711b86d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9\" (UID: \"7d0be4a7-7a63-495e-a4df-560de711b86d\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:45:53.989884 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:53.989863 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rj7g\" (UniqueName: \"kubernetes.io/projected/7d0be4a7-7a63-495e-a4df-560de711b86d-kube-api-access-2rj7g\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9\" (UID: \"7d0be4a7-7a63-495e-a4df-560de711b86d\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:45:54.167237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:54.167211 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:45:54.472669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:54.472640 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9"] Oct 30 09:45:54.473810 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:45:54.473784 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0be4a7_7a63_495e_a4df_560de711b86d.slice/crio-39e57b65acbb92cef27744c7f18e20df52a4a1bb3d477172c66c0d5291b3a87b WatchSource:0}: Error finding container 39e57b65acbb92cef27744c7f18e20df52a4a1bb3d477172c66c0d5291b3a87b: Status 404 returned error can't find the container with id 39e57b65acbb92cef27744c7f18e20df52a4a1bb3d477172c66c0d5291b3a87b Oct 30 09:45:54.782194 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:54.782104 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" event={"ID":"7d0be4a7-7a63-495e-a4df-560de711b86d","Type":"ContainerStarted","Data":"880a65eeb1996d4b9c92ce68b1cbdd3b0e41623658132652e9747621f796cd85"} Oct 30 09:45:54.782194 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:54.782165 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" event={"ID":"7d0be4a7-7a63-495e-a4df-560de711b86d","Type":"ContainerStarted","Data":"39e57b65acbb92cef27744c7f18e20df52a4a1bb3d477172c66c0d5291b3a87b"} Oct 30 09:45:54.782382 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:54.782199 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:45:54.825350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:45:54.825303 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" podStartSLOduration=1.82528734 podStartE2EDuration="1.82528734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:54.822241703 +0000 UTC m=+3072.684947523" watchObservedRunningTime="2026-10-30 09:45:54.82528734 +0000 UTC m=+3072.687993160" Oct 30 09:46:05.794707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:05.794669 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:46:10.778126 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.778092 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95"] Oct 30 09:46:10.778817 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.778802 5938 cpu_manager.go:414] "containerMap: removing container" podUID="41e9f62f-6ced-4207-a09b-b820f0ad142f" containerName="cleanup" Oct 30 09:46:10.778873 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.778818 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e9f62f-6ced-4207-a09b-b820f0ad142f" containerName="cleanup" Oct 30 09:46:10.778873 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.778837 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" containerName="cleanup" Oct 30 09:46:10.778873 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.778842 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" containerName="cleanup" Oct 30 09:46:10.778873 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.778850 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" containerName="cleanup" Oct 30 09:46:10.778873 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.778858 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" containerName="cleanup" Oct 30 09:46:10.778873 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.778867 5938 cpu_manager.go:414] "containerMap: removing container" podUID="fc5fe483-3456-4b2f-b626-a14bb00dac59" containerName="cleanup" Oct 30 09:46:10.778873 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.778872 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5fe483-3456-4b2f-b626-a14bb00dac59" containerName="cleanup" Oct 30 09:46:10.779098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.778883 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7b1877d3-aaac-4287-8450-d02cb2497d3f" containerName="cleanup" Oct 30 09:46:10.779098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.778887 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b1877d3-aaac-4287-8450-d02cb2497d3f" containerName="cleanup" Oct 30 09:46:10.779098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.778915 5938 cpu_manager.go:414] "containerMap: removing container" podUID="40975fdb-ec84-40e3-8b8f-68bfd44c8123" containerName="cleanup" Oct 30 09:46:10.779098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.778919 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40975fdb-ec84-40e3-8b8f-68bfd44c8123" containerName="cleanup" Oct 30 09:46:10.779098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.779023 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="40975fdb-ec84-40e3-8b8f-68bfd44c8123" containerName="cleanup" Oct 30 09:46:10.779098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.779033 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed86ddbe-5980-4dbf-bec8-aa4e8d2270f4" containerName="cleanup" Oct 30 09:46:10.779098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.779040 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30e90f0-9862-48cc-8171-df2f58695096" containerName="cleanup" Oct 30 09:46:10.779098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.779048 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de510d4-441f-4aa1-98d7-c60848cd26ed" containerName="cleanup" Oct 30 09:46:10.779098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.779055 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc6a7ef-c174-4816-938d-817e57691e6b" containerName="cleanup" Oct 30 09:46:10.779098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.779062 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec402ea4-3c3b-4d53-8ffb-aa7443d0a2c4" containerName="cleanup" Oct 30 09:46:10.779098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.779072 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5fe483-3456-4b2f-b626-a14bb00dac59" containerName="cleanup" Oct 30 09:46:10.783277 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.783254 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.795212 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.795184 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br99j\" (UniqueName: \"kubernetes.io/projected/a2a093a8-bad3-4c10-b148-357f42f7910c-kube-api-access-br99j\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.795391 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.795237 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a093a8-bad3-4c10-b148-357f42f7910c-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.795391 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.795335 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a093a8-bad3-4c10-b148-357f42f7910c-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.795529 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.795423 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a093a8-bad3-4c10-b148-357f42f7910c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.795529 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.795486 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a093a8-bad3-4c10-b148-357f42f7910c-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.795529 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.795517 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a093a8-bad3-4c10-b148-357f42f7910c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.795726 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.795560 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a093a8-bad3-4c10-b148-357f42f7910c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.795825 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.795802 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a093a8-bad3-4c10-b148-357f42f7910c-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.795895 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.795841 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a093a8-bad3-4c10-b148-357f42f7910c-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.816116 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.816087 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95"] Oct 30 09:46:10.899263 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.899224 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a093a8-bad3-4c10-b148-357f42f7910c-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.899492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.899286 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a093a8-bad3-4c10-b148-357f42f7910c-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.899492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.899327 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br99j\" (UniqueName: \"kubernetes.io/projected/a2a093a8-bad3-4c10-b148-357f42f7910c-kube-api-access-br99j\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.899492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.899369 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a093a8-bad3-4c10-b148-357f42f7910c-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.899492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.899419 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a093a8-bad3-4c10-b148-357f42f7910c-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.899836 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.899628 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a093a8-bad3-4c10-b148-357f42f7910c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.899836 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.899705 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a093a8-bad3-4c10-b148-357f42f7910c-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.899836 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.899752 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a093a8-bad3-4c10-b148-357f42f7910c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.899836 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.899803 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a093a8-bad3-4c10-b148-357f42f7910c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.903006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.902975 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a093a8-bad3-4c10-b148-357f42f7910c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.903275 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.903253 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a093a8-bad3-4c10-b148-357f42f7910c-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.903507 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.903485 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a093a8-bad3-4c10-b148-357f42f7910c-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.904285 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.904256 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a093a8-bad3-4c10-b148-357f42f7910c-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.907082 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.907053 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a093a8-bad3-4c10-b148-357f42f7910c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.907386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.907356 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a093a8-bad3-4c10-b148-357f42f7910c-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.908105 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.908076 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a093a8-bad3-4c10-b148-357f42f7910c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.924283 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.924263 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a093a8-bad3-4c10-b148-357f42f7910c-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:10.924641 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:10.924617 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br99j\" (UniqueName: \"kubernetes.io/projected/a2a093a8-bad3-4c10-b148-357f42f7910c-kube-api-access-br99j\") pod \"maas-default-gateway-openshift-default-77f556d647-k5n95\" (UID: \"a2a093a8-bad3-4c10-b148-357f42f7910c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:11.102631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:11.102521 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:11.423353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:11.423323 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95"] Oct 30 09:46:11.424167 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:46:11.424128 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a093a8_bad3_4c10_b148_357f42f7910c.slice/crio-b8727fa69301bc062518cfde27b61c1d3d5d029952f6e130c16f0ed6aa9b1160 WatchSource:0}: Error finding container b8727fa69301bc062518cfde27b61c1d3d5d029952f6e130c16f0ed6aa9b1160: Status 404 returned error can't find the container with id b8727fa69301bc062518cfde27b61c1d3d5d029952f6e130c16f0ed6aa9b1160 Oct 30 09:46:11.427890 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:11.427834 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:46:11.428070 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:11.428043 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:46:11.428228 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:11.428111 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:46:11.923035 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:11.922996 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" event={"ID":"a2a093a8-bad3-4c10-b148-357f42f7910c","Type":"ContainerStarted","Data":"3da971bb6dadc821b9718b6a32f39a02ee99a1c6487aa88835956e422026d8dd"} Oct 30 09:46:11.923431 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:11.923055 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" event={"ID":"a2a093a8-bad3-4c10-b148-357f42f7910c","Type":"ContainerStarted","Data":"b8727fa69301bc062518cfde27b61c1d3d5d029952f6e130c16f0ed6aa9b1160"} Oct 30 09:46:11.969192 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:11.969136 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" podStartSLOduration=1.969122344 podStartE2EDuration="1.969122344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:46:11.965819183 +0000 UTC m=+3089.828525003" watchObservedRunningTime="2026-10-30 09:46:11.969122344 +0000 UTC m=+3089.831828168" Oct 30 09:46:12.028887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:12.028858 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s"] Oct 30 09:46:12.033072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:12.033045 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1ae496-hrd5s"] Oct 30 09:46:12.103564 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:12.103535 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:12.105700 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:12.105663 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k5n95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 09:46:12.105872 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:12.105718 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" podUID="a2a093a8-bad3-4c10-b148-357f42f7910c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 09:46:12.259441 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:12.259363 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4" path="/var/lib/kubelet/pods/842dee8f-5b8d-4aca-ae9e-a3f80fe25cd4/volumes" Oct 30 09:46:13.103429 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:13.103293 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k5n95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 09:46:13.103429 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:13.103351 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" podUID="a2a093a8-bad3-4c10-b148-357f42f7910c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 09:46:14.103561 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:14.103533 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k5n95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 09:46:14.103985 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:14.103577 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" podUID="a2a093a8-bad3-4c10-b148-357f42f7910c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 09:46:15.103450 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:15.103422 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k5n95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 09:46:15.103653 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:15.103470 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" podUID="a2a093a8-bad3-4c10-b148-357f42f7910c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 09:46:15.792942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:15.792905 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:46:15.809969 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:46:15.809930 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:46:16.103486 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.103400 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k5n95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 09:46:16.103486 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.103448 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" podUID="a2a093a8-bad3-4c10-b148-357f42f7910c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 09:46:16.351623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.351569 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j"] Oct 30 09:46:16.357130 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.357062 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j"] Oct 30 09:46:16.357294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.357167 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" Oct 30 09:46:16.373890 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.373841 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:46:16.374094 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.374006 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:46:16.415513 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.415486 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf7g8\" (UniqueName: \"kubernetes.io/projected/2d415736-a914-4b4c-941c-0d2315ca7502-kube-api-access-nf7g8\") pod \"maas-api-revoke-sub-685ff309-4kt2j\" (UID: \"2d415736-a914-4b4c-941c-0d2315ca7502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" Oct 30 09:46:16.415708 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.415688 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d415736-a914-4b4c-941c-0d2315ca7502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-685ff309-4kt2j\" (UID: \"2d415736-a914-4b4c-941c-0d2315ca7502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" Oct 30 09:46:16.415793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.415768 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d415736-a914-4b4c-941c-0d2315ca7502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-685ff309-4kt2j\" (UID: \"2d415736-a914-4b4c-941c-0d2315ca7502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" Oct 30 09:46:16.517439 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.517407 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d415736-a914-4b4c-941c-0d2315ca7502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-685ff309-4kt2j\" (UID: \"2d415736-a914-4b4c-941c-0d2315ca7502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" Oct 30 09:46:16.517642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.517543 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf7g8\" (UniqueName: \"kubernetes.io/projected/2d415736-a914-4b4c-941c-0d2315ca7502-kube-api-access-nf7g8\") pod \"maas-api-revoke-sub-685ff309-4kt2j\" (UID: \"2d415736-a914-4b4c-941c-0d2315ca7502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" Oct 30 09:46:16.517698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.517688 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d415736-a914-4b4c-941c-0d2315ca7502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-685ff309-4kt2j\" (UID: \"2d415736-a914-4b4c-941c-0d2315ca7502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" Oct 30 09:46:16.518056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.518036 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d415736-a914-4b4c-941c-0d2315ca7502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-685ff309-4kt2j\" (UID: \"2d415736-a914-4b4c-941c-0d2315ca7502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" Oct 30 09:46:16.545446 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.545380 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d415736-a914-4b4c-941c-0d2315ca7502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-685ff309-4kt2j\" (UID: \"2d415736-a914-4b4c-941c-0d2315ca7502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" Oct 30 09:46:16.548498 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.548470 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf7g8\" (UniqueName: \"kubernetes.io/projected/2d415736-a914-4b4c-941c-0d2315ca7502-kube-api-access-nf7g8\") pod \"maas-api-revoke-sub-685ff309-4kt2j\" (UID: \"2d415736-a914-4b4c-941c-0d2315ca7502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" Oct 30 09:46:16.683799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:16.683767 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" Oct 30 09:46:17.047208 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:17.047177 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j"] Oct 30 09:46:17.048771 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:46:17.048740 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d415736_a914_4b4c_941c_0d2315ca7502.slice/crio-cff593320db024ac23ea1ae1859446322b77f0a595203962bdff06a3f0804757 WatchSource:0}: Error finding container cff593320db024ac23ea1ae1859446322b77f0a595203962bdff06a3f0804757: Status 404 returned error can't find the container with id cff593320db024ac23ea1ae1859446322b77f0a595203962bdff06a3f0804757 Oct 30 09:46:17.104168 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:17.104141 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k5n95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 09:46:17.104478 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:17.104189 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" podUID="a2a093a8-bad3-4c10-b148-357f42f7910c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 09:46:17.986281 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:17.986241 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" event={"ID":"2d415736-a914-4b4c-941c-0d2315ca7502","Type":"ContainerStarted","Data":"b478fada56726373ae3ce95dee061d62f37615881490d26b1da123bd043efc55"} Oct 30 09:46:17.986476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:17.986310 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" event={"ID":"2d415736-a914-4b4c-941c-0d2315ca7502","Type":"ContainerStarted","Data":"cff593320db024ac23ea1ae1859446322b77f0a595203962bdff06a3f0804757"} Oct 30 09:46:18.031269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:18.031222 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" podStartSLOduration=2.031209895 podStartE2EDuration="2.031209895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:46:18.027545173 +0000 UTC m=+3095.890251008" watchObservedRunningTime="2026-10-30 09:46:18.031209895 +0000 UTC m=+3095.893915716" Oct 30 09:46:18.103203 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:18.103173 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k5n95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 09:46:18.103383 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:18.103221 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" podUID="a2a093a8-bad3-4c10-b148-357f42f7910c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 09:46:18.995441 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:18.995409 5938 generic.go:356] "Generic (PLEG): container finished" podID="2d415736-a914-4b4c-941c-0d2315ca7502" containerID="b478fada56726373ae3ce95dee061d62f37615881490d26b1da123bd043efc55" exitCode=0 Oct 30 09:46:18.996020 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:18.995510 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" event={"ID":"2d415736-a914-4b4c-941c-0d2315ca7502","Type":"ContainerDied","Data":"b478fada56726373ae3ce95dee061d62f37615881490d26b1da123bd043efc55"} Oct 30 09:46:19.102998 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:19.102966 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k5n95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 09:46:19.103178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:19.103024 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" podUID="a2a093a8-bad3-4c10-b148-357f42f7910c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 09:46:20.116814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:20.116779 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:20.117812 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:20.117752 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:20.119971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:20.119948 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k5n95" Oct 30 09:46:20.169107 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:20.169088 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" Oct 30 09:46:20.202630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:20.202577 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d415736-a914-4b4c-941c-0d2315ca7502-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d415736-a914-4b4c-941c-0d2315ca7502-maas-api-openshift-ca\") pod \"2d415736-a914-4b4c-941c-0d2315ca7502\" (UID: \"2d415736-a914-4b4c-941c-0d2315ca7502\") " Oct 30 09:46:20.202916 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:20.202685 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d415736-a914-4b4c-941c-0d2315ca7502-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d415736-a914-4b4c-941c-0d2315ca7502-maas-api-xks-ca\") pod \"2d415736-a914-4b4c-941c-0d2315ca7502\" (UID: \"2d415736-a914-4b4c-941c-0d2315ca7502\") " Oct 30 09:46:20.202916 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:20.202881 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d415736-a914-4b4c-941c-0d2315ca7502-kube-api-access-nf7g8\" (UniqueName: \"kubernetes.io/projected/2d415736-a914-4b4c-941c-0d2315ca7502-kube-api-access-nf7g8\") pod \"2d415736-a914-4b4c-941c-0d2315ca7502\" (UID: \"2d415736-a914-4b4c-941c-0d2315ca7502\") " Oct 30 09:46:20.203135 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:20.202917 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d415736-a914-4b4c-941c-0d2315ca7502-maas-api-openshift-ca" pod "2d415736-a914-4b4c-941c-0d2315ca7502" (UID: "2d415736-a914-4b4c-941c-0d2315ca7502"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:46:20.203796 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:20.203772 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d415736-a914-4b4c-941c-0d2315ca7502-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:20.205276 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:20.205254 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d415736-a914-4b4c-941c-0d2315ca7502-kube-api-access-nf7g8" pod "2d415736-a914-4b4c-941c-0d2315ca7502" (UID: "2d415736-a914-4b4c-941c-0d2315ca7502"). InnerVolumeSpecName "kube-api-access-nf7g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:46:20.205369 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:20.205276 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d415736-a914-4b4c-941c-0d2315ca7502-maas-api-xks-ca" pod "2d415736-a914-4b4c-941c-0d2315ca7502" (UID: "2d415736-a914-4b4c-941c-0d2315ca7502"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:46:20.305510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:20.305479 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf7g8\" (UniqueName: \"kubernetes.io/projected/2d415736-a914-4b4c-941c-0d2315ca7502-kube-api-access-nf7g8\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:20.305510 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:20.305512 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d415736-a914-4b4c-941c-0d2315ca7502-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:21.011865 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:21.011836 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" Oct 30 09:46:21.012052 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:21.011865 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j" event={"ID":"2d415736-a914-4b4c-941c-0d2315ca7502","Type":"ContainerDied","Data":"cff593320db024ac23ea1ae1859446322b77f0a595203962bdff06a3f0804757"} Oct 30 09:46:21.012052 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:21.011911 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cff593320db024ac23ea1ae1859446322b77f0a595203962bdff06a3f0804757" Oct 30 09:46:25.793334 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:25.793294 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:46:27.067278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.067246 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t"] Oct 30 09:46:27.068059 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.068042 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2d415736-a914-4b4c-941c-0d2315ca7502" containerName="revoke-keys" Oct 30 09:46:27.068059 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.068060 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d415736-a914-4b4c-941c-0d2315ca7502" containerName="revoke-keys" Oct 30 09:46:27.068186 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.068176 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d415736-a914-4b4c-941c-0d2315ca7502" containerName="revoke-keys" Oct 30 09:46:27.072383 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.072361 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" Oct 30 09:46:27.092618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.092575 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t"] Oct 30 09:46:27.101513 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.101485 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:46:27.101737 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.101630 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:46:27.193389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.193361 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9cx\" (UniqueName: \"kubernetes.io/projected/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-kube-api-access-rs9cx\") pod \"maas-api-revoke-sub-a2e956a3-w282t\" (UID: \"7d2f279e-75fd-424e-8e3f-edd77de0d4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" Oct 30 09:46:27.193560 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.193434 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2e956a3-w282t\" (UID: \"7d2f279e-75fd-424e-8e3f-edd77de0d4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" Oct 30 09:46:27.193708 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.193688 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2e956a3-w282t\" (UID: \"7d2f279e-75fd-424e-8e3f-edd77de0d4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" Oct 30 09:46:27.294368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.294335 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2e956a3-w282t\" (UID: \"7d2f279e-75fd-424e-8e3f-edd77de0d4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" Oct 30 09:46:27.294547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.294498 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9cx\" (UniqueName: \"kubernetes.io/projected/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-kube-api-access-rs9cx\") pod \"maas-api-revoke-sub-a2e956a3-w282t\" (UID: \"7d2f279e-75fd-424e-8e3f-edd77de0d4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" Oct 30 09:46:27.294648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.294573 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2e956a3-w282t\" (UID: \"7d2f279e-75fd-424e-8e3f-edd77de0d4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" Oct 30 09:46:27.295476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.295449 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2e956a3-w282t\" (UID: \"7d2f279e-75fd-424e-8e3f-edd77de0d4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" Oct 30 09:46:27.297000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.296976 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2e956a3-w282t\" (UID: \"7d2f279e-75fd-424e-8e3f-edd77de0d4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" Oct 30 09:46:27.315922 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.315895 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9cx\" (UniqueName: \"kubernetes.io/projected/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-kube-api-access-rs9cx\") pod \"maas-api-revoke-sub-a2e956a3-w282t\" (UID: \"7d2f279e-75fd-424e-8e3f-edd77de0d4a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" Oct 30 09:46:27.387023 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.386995 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" Oct 30 09:46:27.689712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:27.689683 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t"] Oct 30 09:46:27.689941 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:46:27.689918 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2f279e_75fd_424e_8e3f_edd77de0d4a4.slice/crio-8895276efb4997c2c777ca466b0d3460a4925bfdde634221216fd8994c902743 WatchSource:0}: Error finding container 8895276efb4997c2c777ca466b0d3460a4925bfdde634221216fd8994c902743: Status 404 returned error can't find the container with id 8895276efb4997c2c777ca466b0d3460a4925bfdde634221216fd8994c902743 Oct 30 09:46:28.064600 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:28.064561 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" event={"ID":"7d2f279e-75fd-424e-8e3f-edd77de0d4a4","Type":"ContainerStarted","Data":"38269781954c7400108f02c6a1898614e8ad73f7ec06689a12782bfd55dc3e2b"} Oct 30 09:46:28.064731 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:28.064622 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" event={"ID":"7d2f279e-75fd-424e-8e3f-edd77de0d4a4","Type":"ContainerStarted","Data":"8895276efb4997c2c777ca466b0d3460a4925bfdde634221216fd8994c902743"} Oct 30 09:46:28.105657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:28.105578 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" podStartSLOduration=1.105564962 podStartE2EDuration="1.105564962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:46:28.104620644 +0000 UTC m=+3105.967326469" watchObservedRunningTime="2026-10-30 09:46:28.105564962 +0000 UTC m=+3105.968270852" Oct 30 09:46:29.073787 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:29.073701 5938 generic.go:356] "Generic (PLEG): container finished" podID="7d2f279e-75fd-424e-8e3f-edd77de0d4a4" containerID="38269781954c7400108f02c6a1898614e8ad73f7ec06689a12782bfd55dc3e2b" exitCode=0 Oct 30 09:46:29.073940 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:29.073789 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" event={"ID":"7d2f279e-75fd-424e-8e3f-edd77de0d4a4","Type":"ContainerDied","Data":"38269781954c7400108f02c6a1898614e8ad73f7ec06689a12782bfd55dc3e2b"} Oct 30 09:46:30.257669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:30.257646 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" Oct 30 09:46:30.334171 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:30.334087 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-maas-api-xks-ca\") pod \"7d2f279e-75fd-424e-8e3f-edd77de0d4a4\" (UID: \"7d2f279e-75fd-424e-8e3f-edd77de0d4a4\") " Oct 30 09:46:30.334339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:30.334169 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-kube-api-access-rs9cx\" (UniqueName: \"kubernetes.io/projected/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-kube-api-access-rs9cx\") pod \"7d2f279e-75fd-424e-8e3f-edd77de0d4a4\" (UID: \"7d2f279e-75fd-424e-8e3f-edd77de0d4a4\") " Oct 30 09:46:30.334339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:30.334210 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-maas-api-openshift-ca\") pod \"7d2f279e-75fd-424e-8e3f-edd77de0d4a4\" (UID: \"7d2f279e-75fd-424e-8e3f-edd77de0d4a4\") " Oct 30 09:46:30.334720 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:30.334692 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-maas-api-openshift-ca" pod "7d2f279e-75fd-424e-8e3f-edd77de0d4a4" (UID: "7d2f279e-75fd-424e-8e3f-edd77de0d4a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:46:30.335108 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:30.335080 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:30.336581 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:30.336544 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-maas-api-xks-ca" pod "7d2f279e-75fd-424e-8e3f-edd77de0d4a4" (UID: "7d2f279e-75fd-424e-8e3f-edd77de0d4a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:46:30.336712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:30.336694 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-kube-api-access-rs9cx" pod "7d2f279e-75fd-424e-8e3f-edd77de0d4a4" (UID: "7d2f279e-75fd-424e-8e3f-edd77de0d4a4"). InnerVolumeSpecName "kube-api-access-rs9cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:46:30.436396 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:30.436369 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs9cx\" (UniqueName: \"kubernetes.io/projected/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-kube-api-access-rs9cx\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:30.436396 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:30.436395 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2f279e-75fd-424e-8e3f-edd77de0d4a4-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:31.090612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:31.090569 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" Oct 30 09:46:31.090801 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:31.090649 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t" event={"ID":"7d2f279e-75fd-424e-8e3f-edd77de0d4a4","Type":"ContainerDied","Data":"8895276efb4997c2c777ca466b0d3460a4925bfdde634221216fd8994c902743"} Oct 30 09:46:31.090801 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:31.090714 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8895276efb4997c2c777ca466b0d3460a4925bfdde634221216fd8994c902743" Oct 30 09:46:35.793002 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:35.792960 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:46:38.797522 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:38.797487 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s"] Oct 30 09:46:38.798254 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:38.798237 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7d2f279e-75fd-424e-8e3f-edd77de0d4a4" containerName="revoke-keys" Oct 30 09:46:38.798301 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:38.798256 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2f279e-75fd-424e-8e3f-edd77de0d4a4" containerName="revoke-keys" Oct 30 09:46:38.798403 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:38.798394 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2f279e-75fd-424e-8e3f-edd77de0d4a4" containerName="revoke-keys" Oct 30 09:46:38.802454 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:38.802437 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:38.839646 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:38.836370 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s"] Oct 30 09:46:38.964773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:38.964720 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad82bb46-0909-4cbc-af1f-01126e016a9a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s\" (UID: \"ad82bb46-0909-4cbc-af1f-01126e016a9a\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:38.964998 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:38.964790 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sklsp\" (UniqueName: \"kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-kube-api-access-sklsp\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s\" (UID: \"ad82bb46-0909-4cbc-af1f-01126e016a9a\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:38.964998 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:38.964870 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s\" (UID: \"ad82bb46-0909-4cbc-af1f-01126e016a9a\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:38.965125 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:38.964997 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad82bb46-0909-4cbc-af1f-01126e016a9a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s\" (UID: \"ad82bb46-0909-4cbc-af1f-01126e016a9a\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:39.066045 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:39.065951 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sklsp\" (UniqueName: \"kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-kube-api-access-sklsp\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s\" (UID: \"ad82bb46-0909-4cbc-af1f-01126e016a9a\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:39.066045 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:39.066025 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s\" (UID: \"ad82bb46-0909-4cbc-af1f-01126e016a9a\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:39.066342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:39.066118 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad82bb46-0909-4cbc-af1f-01126e016a9a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s\" (UID: \"ad82bb46-0909-4cbc-af1f-01126e016a9a\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:39.066342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:39.066243 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad82bb46-0909-4cbc-af1f-01126e016a9a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s\" (UID: \"ad82bb46-0909-4cbc-af1f-01126e016a9a\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:39.066915 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:39.066896 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad82bb46-0909-4cbc-af1f-01126e016a9a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s\" (UID: \"ad82bb46-0909-4cbc-af1f-01126e016a9a\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:39.067028 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:39.067008 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad82bb46-0909-4cbc-af1f-01126e016a9a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s\" (UID: \"ad82bb46-0909-4cbc-af1f-01126e016a9a\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:39.068819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:39.068796 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s\" (UID: \"ad82bb46-0909-4cbc-af1f-01126e016a9a\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:39.091122 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:39.091088 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sklsp\" (UniqueName: \"kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-kube-api-access-sklsp\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s\" (UID: \"ad82bb46-0909-4cbc-af1f-01126e016a9a\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:39.115899 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:39.115874 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:39.422954 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:39.422921 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s"] Oct 30 09:46:40.090820 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.090789 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv"] Oct 30 09:46:40.095408 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.095383 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" Oct 30 09:46:40.117151 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.117123 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv"] Oct 30 09:46:40.125798 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.125764 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:46:40.125946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.125837 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:46:40.163389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.163359 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" event={"ID":"ad82bb46-0909-4cbc-af1f-01126e016a9a","Type":"ContainerStarted","Data":"4bca0d3856947f7ea1ae693f342ef1a828837a795633ad5115c12682d5108720"} Oct 30 09:46:40.163558 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.163413 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" event={"ID":"ad82bb46-0909-4cbc-af1f-01126e016a9a","Type":"ContainerStarted","Data":"311bbe6b16a146f75cb381dd29834bc20253b704f12358ee23f92c1ce4cc09d9"} Oct 30 09:46:40.163558 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.163436 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:40.203572 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.203546 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc4x4\" (UniqueName: \"kubernetes.io/projected/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-kube-api-access-vc4x4\") pod \"maas-api-revoke-sub-00702f57-tv6rv\" (UID: \"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" Oct 30 09:46:40.203717 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.203695 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00702f57-tv6rv\" (UID: \"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" Oct 30 09:46:40.203845 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.203829 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00702f57-tv6rv\" (UID: \"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" Oct 30 09:46:40.204180 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:46:40.204153 5938 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-881f83/agc-cred-fa5ab9b30048: secret "agc-cred-fa5ab9b30048" not found Oct 30 09:46:40.204180 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:46:40.204177 5938 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s: secret "agc-cred-fa5ab9b30048" not found Oct 30 09:46:40.204313 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:46:40.204252 5938 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-provider-credentials podName:ad82bb46-0909-4cbc-af1f-01126e016a9a nodeName:}" failed. No retries permitted until 2026-10-30 09:46:40.704233631 +0000 UTC m=+3118.566939465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" (UID: "ad82bb46-0909-4cbc-af1f-01126e016a9a") : secret "agc-cred-fa5ab9b30048" not found Oct 30 09:46:40.207217 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.207177 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" podStartSLOduration=2.207165172 podStartE2EDuration="2.207165172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:46:40.204710579 +0000 UTC m=+3118.067416402" watchObservedRunningTime="2026-10-30 09:46:40.207165172 +0000 UTC m=+3118.069871014" Oct 30 09:46:40.305067 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.305043 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00702f57-tv6rv\" (UID: \"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" Oct 30 09:46:40.305222 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.305187 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc4x4\" (UniqueName: \"kubernetes.io/projected/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-kube-api-access-vc4x4\") pod \"maas-api-revoke-sub-00702f57-tv6rv\" (UID: \"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" Oct 30 09:46:40.305313 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.305224 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00702f57-tv6rv\" (UID: \"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" Oct 30 09:46:40.305750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.305727 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00702f57-tv6rv\" (UID: \"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" Oct 30 09:46:40.307667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.307651 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00702f57-tv6rv\" (UID: \"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" Oct 30 09:46:40.325870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.325850 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc4x4\" (UniqueName: \"kubernetes.io/projected/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-kube-api-access-vc4x4\") pod \"maas-api-revoke-sub-00702f57-tv6rv\" (UID: \"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" Oct 30 09:46:40.411999 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.411975 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" Oct 30 09:46:40.622161 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.622124 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s"] Oct 30 09:46:40.710240 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:46:40.710173 5938 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-881f83/agc-cred-fa5ab9b30048: secret "agc-cred-fa5ab9b30048" not found Oct 30 09:46:40.710240 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:46:40.710195 5938 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s: secret "agc-cred-fa5ab9b30048" not found Oct 30 09:46:40.710416 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:46:40.710245 5938 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-provider-credentials podName:ad82bb46-0909-4cbc-af1f-01126e016a9a nodeName:}" failed. No retries permitted until 2026-10-30 09:46:41.710230357 +0000 UTC m=+3119.572936159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" (UID: "ad82bb46-0909-4cbc-af1f-01126e016a9a") : secret "agc-cred-fa5ab9b30048" not found Oct 30 09:46:40.718693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:40.718669 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv"] Oct 30 09:46:40.719876 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:46:40.719847 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aeaf185_a46f_4b3c_a52d_8c97f4733dd6.slice/crio-8c64e9ae16948e256b32b8e1cc7728cc24d1010948a0d663832739fb60aecee1 WatchSource:0}: Error finding container 8c64e9ae16948e256b32b8e1cc7728cc24d1010948a0d663832739fb60aecee1: Status 404 returned error can't find the container with id 8c64e9ae16948e256b32b8e1cc7728cc24d1010948a0d663832739fb60aecee1 Oct 30 09:46:41.172946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:41.172907 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" event={"ID":"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6","Type":"ContainerStarted","Data":"baac80030ec2bc55a14ecf37a413d2062f175761bb3e1cbeee2c602a1c7bd585"} Oct 30 09:46:41.173378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:41.172967 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" event={"ID":"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6","Type":"ContainerStarted","Data":"8c64e9ae16948e256b32b8e1cc7728cc24d1010948a0d663832739fb60aecee1"} Oct 30 09:46:41.216314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:41.216274 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" podStartSLOduration=1.2162605659999999 podStartE2EDuration="1.216260566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:46:41.213742244 +0000 UTC m=+3119.076448064" watchObservedRunningTime="2026-10-30 09:46:41.216260566 +0000 UTC m=+3119.078966387" Oct 30 09:46:41.724505 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:46:41.724480 5938 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-881f83/agc-cred-fa5ab9b30048: secret "agc-cred-fa5ab9b30048" not found Oct 30 09:46:41.724505 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:46:41.724507 5938 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s: secret "agc-cred-fa5ab9b30048" not found Oct 30 09:46:41.724793 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:46:41.724578 5938 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-provider-credentials podName:ad82bb46-0909-4cbc-af1f-01126e016a9a nodeName:}" failed. No retries permitted until 2026-10-30 09:46:43.724556245 +0000 UTC m=+3121.587262063 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" (UID: "ad82bb46-0909-4cbc-af1f-01126e016a9a") : secret "agc-cred-fa5ab9b30048" not found Oct 30 09:46:42.182455 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:42.182414 5938 generic.go:356] "Generic (PLEG): container finished" podID="9aeaf185-a46f-4b3c-a52d-8c97f4733dd6" containerID="baac80030ec2bc55a14ecf37a413d2062f175761bb3e1cbeee2c602a1c7bd585" exitCode=0 Oct 30 09:46:42.182854 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:42.182491 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" event={"ID":"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6","Type":"ContainerDied","Data":"baac80030ec2bc55a14ecf37a413d2062f175761bb3e1cbeee2c602a1c7bd585"} Oct 30 09:46:43.351750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:43.351716 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" Oct 30 09:46:43.446062 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:43.445987 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-maas-api-xks-ca\") pod \"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6\" (UID: \"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6\") " Oct 30 09:46:43.446212 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:43.446118 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-kube-api-access-vc4x4\" (UniqueName: \"kubernetes.io/projected/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-kube-api-access-vc4x4\") pod \"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6\" (UID: \"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6\") " Oct 30 09:46:43.446303 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:43.446286 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-maas-api-openshift-ca\") pod \"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6\" (UID: \"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6\") " Oct 30 09:46:43.446708 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:43.446679 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-maas-api-openshift-ca" pod "9aeaf185-a46f-4b3c-a52d-8c97f4733dd6" (UID: "9aeaf185-a46f-4b3c-a52d-8c97f4733dd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:46:43.447007 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:43.446961 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:43.448182 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:43.448158 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-maas-api-xks-ca" pod "9aeaf185-a46f-4b3c-a52d-8c97f4733dd6" (UID: "9aeaf185-a46f-4b3c-a52d-8c97f4733dd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:46:43.448373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:43.448354 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-kube-api-access-vc4x4" pod "9aeaf185-a46f-4b3c-a52d-8c97f4733dd6" (UID: "9aeaf185-a46f-4b3c-a52d-8c97f4733dd6"). InnerVolumeSpecName "kube-api-access-vc4x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:46:43.547971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:43.547943 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc4x4\" (UniqueName: \"kubernetes.io/projected/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-kube-api-access-vc4x4\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:43.548116 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:43.547973 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:43.750972 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:46:43.750903 5938 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-881f83/agc-cred-fa5ab9b30048: secret "agc-cred-fa5ab9b30048" not found Oct 30 09:46:43.750972 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:46:43.750927 5938 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s: secret "agc-cred-fa5ab9b30048" not found Oct 30 09:46:43.751167 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:46:43.750991 5938 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-provider-credentials podName:ad82bb46-0909-4cbc-af1f-01126e016a9a nodeName:}" failed. No retries permitted until 2026-10-30 09:46:47.750971902 +0000 UTC m=+3125.613677707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" (UID: "ad82bb46-0909-4cbc-af1f-01126e016a9a") : secret "agc-cred-fa5ab9b30048" not found Oct 30 09:46:44.200158 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:44.200122 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" event={"ID":"9aeaf185-a46f-4b3c-a52d-8c97f4733dd6","Type":"ContainerDied","Data":"8c64e9ae16948e256b32b8e1cc7728cc24d1010948a0d663832739fb60aecee1"} Oct 30 09:46:44.200351 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:44.200170 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c64e9ae16948e256b32b8e1cc7728cc24d1010948a0d663832739fb60aecee1" Oct 30 09:46:44.200351 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:44.200135 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv" Oct 30 09:46:45.793788 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:45.793745 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:46:45.809807 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:46:45.809759 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:46:47.183415 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:47.183361 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" podUID="ad82bb46-0909-4cbc-af1f-01126e016a9a" containerName="payload-processing" containerID="cri-o://4bca0d3856947f7ea1ae693f342ef1a828837a795633ad5115c12682d5108720" gracePeriod=25 Oct 30 09:46:47.464844 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:47.464820 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:47.599977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:47.599942 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad82bb46-0909-4cbc-af1f-01126e016a9a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad82bb46-0909-4cbc-af1f-01126e016a9a-routing-overlay-volume\") pod \"ad82bb46-0909-4cbc-af1f-01126e016a9a\" (UID: \"ad82bb46-0909-4cbc-af1f-01126e016a9a\") " Oct 30 09:46:47.600177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:47.600018 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-kube-api-access-sklsp\" (UniqueName: \"kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-kube-api-access-sklsp\") pod \"ad82bb46-0909-4cbc-af1f-01126e016a9a\" (UID: \"ad82bb46-0909-4cbc-af1f-01126e016a9a\") " Oct 30 09:46:47.600177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:47.600084 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-provider-credentials\") pod \"ad82bb46-0909-4cbc-af1f-01126e016a9a\" (UID: \"ad82bb46-0909-4cbc-af1f-01126e016a9a\") " Oct 30 09:46:47.600355 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:47.600339 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad82bb46-0909-4cbc-af1f-01126e016a9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad82bb46-0909-4cbc-af1f-01126e016a9a-plugins-config-volume\") pod \"ad82bb46-0909-4cbc-af1f-01126e016a9a\" (UID: \"ad82bb46-0909-4cbc-af1f-01126e016a9a\") " Oct 30 09:46:47.600447 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:47.600333 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad82bb46-0909-4cbc-af1f-01126e016a9a-routing-overlay-volume" pod "ad82bb46-0909-4cbc-af1f-01126e016a9a" (UID: "ad82bb46-0909-4cbc-af1f-01126e016a9a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:46:47.600729 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:47.600690 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad82bb46-0909-4cbc-af1f-01126e016a9a-plugins-config-volume" pod "ad82bb46-0909-4cbc-af1f-01126e016a9a" (UID: "ad82bb46-0909-4cbc-af1f-01126e016a9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:46:47.601050 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:47.601032 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad82bb46-0909-4cbc-af1f-01126e016a9a-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:47.601107 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:47.601056 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad82bb46-0909-4cbc-af1f-01126e016a9a-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:47.603015 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:47.602995 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-provider-credentials" pod "ad82bb46-0909-4cbc-af1f-01126e016a9a" (UID: "ad82bb46-0909-4cbc-af1f-01126e016a9a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:46:47.603108 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:47.603026 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-kube-api-access-sklsp" pod "ad82bb46-0909-4cbc-af1f-01126e016a9a" (UID: "ad82bb46-0909-4cbc-af1f-01126e016a9a"). InnerVolumeSpecName "kube-api-access-sklsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:46:47.702837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:47.702768 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sklsp\" (UniqueName: \"kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-kube-api-access-sklsp\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:47.702837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:47.702791 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad82bb46-0909-4cbc-af1f-01126e016a9a-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:48.231720 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:48.231688 5938 generic.go:356] "Generic (PLEG): container finished" podID="ad82bb46-0909-4cbc-af1f-01126e016a9a" containerID="4bca0d3856947f7ea1ae693f342ef1a828837a795633ad5115c12682d5108720" exitCode=0 Oct 30 09:46:48.232140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:48.231742 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" Oct 30 09:46:48.232140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:48.231762 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" event={"ID":"ad82bb46-0909-4cbc-af1f-01126e016a9a","Type":"ContainerDied","Data":"4bca0d3856947f7ea1ae693f342ef1a828837a795633ad5115c12682d5108720"} Oct 30 09:46:48.232140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:48.231817 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s" event={"ID":"ad82bb46-0909-4cbc-af1f-01126e016a9a","Type":"ContainerDied","Data":"311bbe6b16a146f75cb381dd29834bc20253b704f12358ee23f92c1ce4cc09d9"} Oct 30 09:46:48.232140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:48.231847 5938 scope.go:122] "RemoveContainer" containerID="4bca0d3856947f7ea1ae693f342ef1a828837a795633ad5115c12682d5108720" Oct 30 09:46:48.243701 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:48.243681 5938 scope.go:122] "RemoveContainer" containerID="4bca0d3856947f7ea1ae693f342ef1a828837a795633ad5115c12682d5108720" Oct 30 09:46:48.243996 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:46:48.243977 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bca0d3856947f7ea1ae693f342ef1a828837a795633ad5115c12682d5108720\": container with ID starting with 4bca0d3856947f7ea1ae693f342ef1a828837a795633ad5115c12682d5108720 not found: ID does not exist" containerID="4bca0d3856947f7ea1ae693f342ef1a828837a795633ad5115c12682d5108720" Oct 30 09:46:48.244046 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:48.244006 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bca0d3856947f7ea1ae693f342ef1a828837a795633ad5115c12682d5108720"} err="failed to get container status \"4bca0d3856947f7ea1ae693f342ef1a828837a795633ad5115c12682d5108720\": rpc error: code = NotFound desc = could not find container \"4bca0d3856947f7ea1ae693f342ef1a828837a795633ad5115c12682d5108720\": container with ID starting with 4bca0d3856947f7ea1ae693f342ef1a828837a795633ad5115c12682d5108720 not found: ID does not exist" Oct 30 09:46:48.286787 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:48.286765 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s"] Oct 30 09:46:48.299857 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:48.299833 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5c52f4s"] Oct 30 09:46:50.265953 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:50.265913 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad82bb46-0909-4cbc-af1f-01126e016a9a" path="/var/lib/kubelet/pods/ad82bb46-0909-4cbc-af1f-01126e016a9a/volumes" Oct 30 09:46:52.858540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:52.858509 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv"] Oct 30 09:46:55.792877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:55.792836 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:46:57.859004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:57.858928 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" podUID="1bea1a86-26a1-4395-84dd-9c94c9e6413b" containerName="payload-processing" containerID="cri-o://52a4d5c99e5fde5e8d22aac33bdba6a289c7bec846f219dd7fac58c3a5797e12" gracePeriod=25 Oct 30 09:46:58.139812 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.139790 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:46:58.253337 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.253302 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bea1a86-26a1-4395-84dd-9c94c9e6413b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bea1a86-26a1-4395-84dd-9c94c9e6413b-provider-credentials\") pod \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\" (UID: \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\") " Oct 30 09:46:58.253557 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.253380 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bea1a86-26a1-4395-84dd-9c94c9e6413b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bea1a86-26a1-4395-84dd-9c94c9e6413b-plugins-config-volume\") pod \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\" (UID: \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\") " Oct 30 09:46:58.253823 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.253788 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bea1a86-26a1-4395-84dd-9c94c9e6413b-kube-api-access-nzwxw\" (UniqueName: \"kubernetes.io/projected/1bea1a86-26a1-4395-84dd-9c94c9e6413b-kube-api-access-nzwxw\") pod \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\" (UID: \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\") " Oct 30 09:46:58.254363 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.253949 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bea1a86-26a1-4395-84dd-9c94c9e6413b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bea1a86-26a1-4395-84dd-9c94c9e6413b-routing-overlay-volume\") pod \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\" (UID: \"1bea1a86-26a1-4395-84dd-9c94c9e6413b\") " Oct 30 09:46:58.254363 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.254065 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bea1a86-26a1-4395-84dd-9c94c9e6413b-plugins-config-volume" pod "1bea1a86-26a1-4395-84dd-9c94c9e6413b" (UID: "1bea1a86-26a1-4395-84dd-9c94c9e6413b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:46:58.254363 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.254324 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bea1a86-26a1-4395-84dd-9c94c9e6413b-routing-overlay-volume" pod "1bea1a86-26a1-4395-84dd-9c94c9e6413b" (UID: "1bea1a86-26a1-4395-84dd-9c94c9e6413b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:46:58.254962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.254942 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bea1a86-26a1-4395-84dd-9c94c9e6413b-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:58.255061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.254969 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bea1a86-26a1-4395-84dd-9c94c9e6413b-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:58.256305 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.256284 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bea1a86-26a1-4395-84dd-9c94c9e6413b-provider-credentials" pod "1bea1a86-26a1-4395-84dd-9c94c9e6413b" (UID: "1bea1a86-26a1-4395-84dd-9c94c9e6413b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:46:58.256693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.256669 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bea1a86-26a1-4395-84dd-9c94c9e6413b-kube-api-access-nzwxw" pod "1bea1a86-26a1-4395-84dd-9c94c9e6413b" (UID: "1bea1a86-26a1-4395-84dd-9c94c9e6413b"). InnerVolumeSpecName "kube-api-access-nzwxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:46:58.321786 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.321760 5938 generic.go:356] "Generic (PLEG): container finished" podID="1bea1a86-26a1-4395-84dd-9c94c9e6413b" containerID="52a4d5c99e5fde5e8d22aac33bdba6a289c7bec846f219dd7fac58c3a5797e12" exitCode=0 Oct 30 09:46:58.321918 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.321808 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" Oct 30 09:46:58.321918 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.321847 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" event={"ID":"1bea1a86-26a1-4395-84dd-9c94c9e6413b","Type":"ContainerDied","Data":"52a4d5c99e5fde5e8d22aac33bdba6a289c7bec846f219dd7fac58c3a5797e12"} Oct 30 09:46:58.321918 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.321899 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv" event={"ID":"1bea1a86-26a1-4395-84dd-9c94c9e6413b","Type":"ContainerDied","Data":"c6db529e34228e484925c9282d19de7f3170a7e492490a1fd31f0dc368867262"} Oct 30 09:46:58.322082 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.321924 5938 scope.go:122] "RemoveContainer" containerID="52a4d5c99e5fde5e8d22aac33bdba6a289c7bec846f219dd7fac58c3a5797e12" Oct 30 09:46:58.332430 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.332409 5938 scope.go:122] "RemoveContainer" containerID="52a4d5c99e5fde5e8d22aac33bdba6a289c7bec846f219dd7fac58c3a5797e12" Oct 30 09:46:58.332746 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:46:58.332727 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52a4d5c99e5fde5e8d22aac33bdba6a289c7bec846f219dd7fac58c3a5797e12\": container with ID starting with 52a4d5c99e5fde5e8d22aac33bdba6a289c7bec846f219dd7fac58c3a5797e12 not found: ID does not exist" containerID="52a4d5c99e5fde5e8d22aac33bdba6a289c7bec846f219dd7fac58c3a5797e12" Oct 30 09:46:58.332832 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.332752 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52a4d5c99e5fde5e8d22aac33bdba6a289c7bec846f219dd7fac58c3a5797e12"} err="failed to get container status \"52a4d5c99e5fde5e8d22aac33bdba6a289c7bec846f219dd7fac58c3a5797e12\": rpc error: code = NotFound desc = could not find container \"52a4d5c99e5fde5e8d22aac33bdba6a289c7bec846f219dd7fac58c3a5797e12\": container with ID starting with 52a4d5c99e5fde5e8d22aac33bdba6a289c7bec846f219dd7fac58c3a5797e12 not found: ID does not exist" Oct 30 09:46:58.356083 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.356052 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bea1a86-26a1-4395-84dd-9c94c9e6413b-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:58.356222 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.356086 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzwxw\" (UniqueName: \"kubernetes.io/projected/1bea1a86-26a1-4395-84dd-9c94c9e6413b-kube-api-access-nzwxw\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:46:58.383021 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.382942 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv"] Oct 30 09:46:58.388178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:46:58.388156 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-7fdfjpmbv"] Oct 30 09:47:00.258750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:00.258718 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bea1a86-26a1-4395-84dd-9c94c9e6413b" path="/var/lib/kubelet/pods/1bea1a86-26a1-4395-84dd-9c94c9e6413b/volumes" Oct 30 09:47:02.256103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:02.256064 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:47:07.757155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.757121 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v"] Oct 30 09:47:07.757913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.757894 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9aeaf185-a46f-4b3c-a52d-8c97f4733dd6" containerName="revoke-keys" Oct 30 09:47:07.757913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.757913 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aeaf185-a46f-4b3c-a52d-8c97f4733dd6" containerName="revoke-keys" Oct 30 09:47:07.758065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.757963 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1bea1a86-26a1-4395-84dd-9c94c9e6413b" containerName="payload-processing" Oct 30 09:47:07.758065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.757970 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bea1a86-26a1-4395-84dd-9c94c9e6413b" containerName="payload-processing" Oct 30 09:47:07.758065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.757991 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ad82bb46-0909-4cbc-af1f-01126e016a9a" containerName="payload-processing" Oct 30 09:47:07.758065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.757996 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad82bb46-0909-4cbc-af1f-01126e016a9a" containerName="payload-processing" Oct 30 09:47:07.758194 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.758107 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bea1a86-26a1-4395-84dd-9c94c9e6413b" containerName="payload-processing" Oct 30 09:47:07.758194 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.758125 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad82bb46-0909-4cbc-af1f-01126e016a9a" containerName="payload-processing" Oct 30 09:47:07.758194 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.758136 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aeaf185-a46f-4b3c-a52d-8c97f4733dd6" containerName="revoke-keys" Oct 30 09:47:07.763208 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.763190 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:47:07.781780 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.781747 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v"] Oct 30 09:47:07.790899 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.790873 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-908e572dbf63\"" Oct 30 09:47:07.887907 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.887874 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb642f5e-4496-480d-a324-06c0df2fe3bb-routing-overlay-volume\") pod \"payload-processing-external-model-58fbcd848f-ssc8v\" (UID: \"eb642f5e-4496-480d-a324-06c0df2fe3bb\") " pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:47:07.888111 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.888090 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb642f5e-4496-480d-a324-06c0df2fe3bb-plugins-config-volume\") pod \"payload-processing-external-model-58fbcd848f-ssc8v\" (UID: \"eb642f5e-4496-480d-a324-06c0df2fe3bb\") " pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:47:07.888331 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.888309 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb642f5e-4496-480d-a324-06c0df2fe3bb-provider-credentials\") pod \"payload-processing-external-model-58fbcd848f-ssc8v\" (UID: \"eb642f5e-4496-480d-a324-06c0df2fe3bb\") " pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:47:07.888398 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.888376 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8tql\" (UniqueName: \"kubernetes.io/projected/eb642f5e-4496-480d-a324-06c0df2fe3bb-kube-api-access-j8tql\") pod \"payload-processing-external-model-58fbcd848f-ssc8v\" (UID: \"eb642f5e-4496-480d-a324-06c0df2fe3bb\") " pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:47:07.989952 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.989925 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb642f5e-4496-480d-a324-06c0df2fe3bb-provider-credentials\") pod \"payload-processing-external-model-58fbcd848f-ssc8v\" (UID: \"eb642f5e-4496-480d-a324-06c0df2fe3bb\") " pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:47:07.990108 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.989967 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8tql\" (UniqueName: \"kubernetes.io/projected/eb642f5e-4496-480d-a324-06c0df2fe3bb-kube-api-access-j8tql\") pod \"payload-processing-external-model-58fbcd848f-ssc8v\" (UID: \"eb642f5e-4496-480d-a324-06c0df2fe3bb\") " pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:47:07.990108 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.990008 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb642f5e-4496-480d-a324-06c0df2fe3bb-routing-overlay-volume\") pod \"payload-processing-external-model-58fbcd848f-ssc8v\" (UID: \"eb642f5e-4496-480d-a324-06c0df2fe3bb\") " pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:47:07.990108 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.990083 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb642f5e-4496-480d-a324-06c0df2fe3bb-plugins-config-volume\") pod \"payload-processing-external-model-58fbcd848f-ssc8v\" (UID: \"eb642f5e-4496-480d-a324-06c0df2fe3bb\") " pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:47:07.990815 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.990739 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb642f5e-4496-480d-a324-06c0df2fe3bb-routing-overlay-volume\") pod \"payload-processing-external-model-58fbcd848f-ssc8v\" (UID: \"eb642f5e-4496-480d-a324-06c0df2fe3bb\") " pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:47:07.990815 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.990812 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb642f5e-4496-480d-a324-06c0df2fe3bb-plugins-config-volume\") pod \"payload-processing-external-model-58fbcd848f-ssc8v\" (UID: \"eb642f5e-4496-480d-a324-06c0df2fe3bb\") " pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:47:07.992791 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:07.992771 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb642f5e-4496-480d-a324-06c0df2fe3bb-provider-credentials\") pod \"payload-processing-external-model-58fbcd848f-ssc8v\" (UID: \"eb642f5e-4496-480d-a324-06c0df2fe3bb\") " pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:47:08.010318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:08.010263 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8tql\" (UniqueName: \"kubernetes.io/projected/eb642f5e-4496-480d-a324-06c0df2fe3bb-kube-api-access-j8tql\") pod \"payload-processing-external-model-58fbcd848f-ssc8v\" (UID: \"eb642f5e-4496-480d-a324-06c0df2fe3bb\") " pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:47:08.078131 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:08.078110 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:47:08.388244 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:47:08.388213 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb642f5e_4496_480d_a324_06c0df2fe3bb.slice/crio-0e8841cb2b75e43345fbbd29b895220044d589b8f166d196d6479f30cb71e274 WatchSource:0}: Error finding container 0e8841cb2b75e43345fbbd29b895220044d589b8f166d196d6479f30cb71e274: Status 404 returned error can't find the container with id 0e8841cb2b75e43345fbbd29b895220044d589b8f166d196d6479f30cb71e274 Oct 30 09:47:08.388446 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:08.388420 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v"] Oct 30 09:47:08.409466 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:08.409430 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" event={"ID":"eb642f5e-4496-480d-a324-06c0df2fe3bb","Type":"ContainerStarted","Data":"0e8841cb2b75e43345fbbd29b895220044d589b8f166d196d6479f30cb71e274"} Oct 30 09:47:09.418893 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:09.418842 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" event={"ID":"eb642f5e-4496-480d-a324-06c0df2fe3bb","Type":"ContainerStarted","Data":"5fcd684aa0e4314ce89f2b1c906e5d38fcd9a8af5a95f5343563a4afb6266a70"} Oct 30 09:47:09.419340 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:09.418963 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:47:09.460478 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:09.460435 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" podStartSLOduration=2.460422904 podStartE2EDuration="2.460422904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:47:09.458793285 +0000 UTC m=+3147.321499108" watchObservedRunningTime="2026-10-30 09:47:09.460422904 +0000 UTC m=+3147.323128724" Oct 30 09:47:12.255711 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:12.255677 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:47:15.809564 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:47:15.809523 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:47:20.429005 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:20.428972 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:47:20.502103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:20.502071 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-74f944f77-jfq98"] Oct 30 09:47:22.256119 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:22.256036 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:47:23.029233 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:23.029195 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r"] Oct 30 09:47:23.042250 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:23.042225 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32eede41-djn2r"] Oct 30 09:47:24.289455 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:24.289420 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dea5010-8500-425d-832a-e582e6258d00" path="/var/lib/kubelet/pods/3dea5010-8500-425d-832a-e582e6258d00/volumes" Oct 30 09:47:25.503455 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:25.503399 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" podUID="a6d0da42-776d-4a4a-980d-c2494e623bc9" containerName="payload-processing" containerID="cri-o://54e2fa9d516e4d2c8bbf0149a68268a445d89aa2adaff3428348829a742bba00" gracePeriod=25 Oct 30 09:47:25.858394 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:25.858367 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:47:25.944004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:25.943968 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d0da42-776d-4a4a-980d-c2494e623bc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d0da42-776d-4a4a-980d-c2494e623bc9-plugins-config-volume\") pod \"a6d0da42-776d-4a4a-980d-c2494e623bc9\" (UID: \"a6d0da42-776d-4a4a-980d-c2494e623bc9\") " Oct 30 09:47:25.946015 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:25.945992 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d0da42-776d-4a4a-980d-c2494e623bc9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6d0da42-776d-4a4a-980d-c2494e623bc9-provider-credentials\") pod \"a6d0da42-776d-4a4a-980d-c2494e623bc9\" (UID: \"a6d0da42-776d-4a4a-980d-c2494e623bc9\") " Oct 30 09:47:25.946268 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:25.946250 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d0da42-776d-4a4a-980d-c2494e623bc9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6d0da42-776d-4a4a-980d-c2494e623bc9-routing-overlay-volume\") pod \"a6d0da42-776d-4a4a-980d-c2494e623bc9\" (UID: \"a6d0da42-776d-4a4a-980d-c2494e623bc9\") " Oct 30 09:47:25.946940 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:25.946904 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d0da42-776d-4a4a-980d-c2494e623bc9-kube-api-access-z2nwb\" (UniqueName: \"kubernetes.io/projected/a6d0da42-776d-4a4a-980d-c2494e623bc9-kube-api-access-z2nwb\") pod \"a6d0da42-776d-4a4a-980d-c2494e623bc9\" (UID: \"a6d0da42-776d-4a4a-980d-c2494e623bc9\") " Oct 30 09:47:25.948162 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:25.944574 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d0da42-776d-4a4a-980d-c2494e623bc9-plugins-config-volume" pod "a6d0da42-776d-4a4a-980d-c2494e623bc9" (UID: "a6d0da42-776d-4a4a-980d-c2494e623bc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:47:25.948318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:25.946733 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d0da42-776d-4a4a-980d-c2494e623bc9-routing-overlay-volume" pod "a6d0da42-776d-4a4a-980d-c2494e623bc9" (UID: "a6d0da42-776d-4a4a-980d-c2494e623bc9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:47:25.950887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:25.950853 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d0da42-776d-4a4a-980d-c2494e623bc9-kube-api-access-z2nwb" pod "a6d0da42-776d-4a4a-980d-c2494e623bc9" (UID: "a6d0da42-776d-4a4a-980d-c2494e623bc9"). InnerVolumeSpecName "kube-api-access-z2nwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:47:25.965084 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:25.965055 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d0da42-776d-4a4a-980d-c2494e623bc9-provider-credentials" pod "a6d0da42-776d-4a4a-980d-c2494e623bc9" (UID: "a6d0da42-776d-4a4a-980d-c2494e623bc9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:47:26.050353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:26.050317 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d0da42-776d-4a4a-980d-c2494e623bc9-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:26.050538 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:26.050359 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6d0da42-776d-4a4a-980d-c2494e623bc9-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:26.050538 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:26.050378 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6d0da42-776d-4a4a-980d-c2494e623bc9-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:26.050538 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:26.050394 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2nwb\" (UniqueName: \"kubernetes.io/projected/a6d0da42-776d-4a4a-980d-c2494e623bc9-kube-api-access-z2nwb\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:26.602237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:26.602052 5938 generic.go:356] "Generic (PLEG): container finished" podID="a6d0da42-776d-4a4a-980d-c2494e623bc9" containerID="54e2fa9d516e4d2c8bbf0149a68268a445d89aa2adaff3428348829a742bba00" exitCode=0 Oct 30 09:47:26.602237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:26.602104 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" Oct 30 09:47:26.602237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:26.602140 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" event={"ID":"a6d0da42-776d-4a4a-980d-c2494e623bc9","Type":"ContainerDied","Data":"54e2fa9d516e4d2c8bbf0149a68268a445d89aa2adaff3428348829a742bba00"} Oct 30 09:47:26.602852 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:26.602653 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74f944f77-jfq98" event={"ID":"a6d0da42-776d-4a4a-980d-c2494e623bc9","Type":"ContainerDied","Data":"022209142ec8cc11ad30a976899ad0ecd0744ef723db8e7773d3bca2579f83f9"} Oct 30 09:47:26.602852 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:26.602721 5938 scope.go:122] "RemoveContainer" containerID="54e2fa9d516e4d2c8bbf0149a68268a445d89aa2adaff3428348829a742bba00" Oct 30 09:47:26.615145 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:26.615124 5938 scope.go:122] "RemoveContainer" containerID="54e2fa9d516e4d2c8bbf0149a68268a445d89aa2adaff3428348829a742bba00" Oct 30 09:47:26.615440 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:47:26.615410 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e2fa9d516e4d2c8bbf0149a68268a445d89aa2adaff3428348829a742bba00\": container with ID starting with 54e2fa9d516e4d2c8bbf0149a68268a445d89aa2adaff3428348829a742bba00 not found: ID does not exist" containerID="54e2fa9d516e4d2c8bbf0149a68268a445d89aa2adaff3428348829a742bba00" Oct 30 09:47:26.615662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:26.615570 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e2fa9d516e4d2c8bbf0149a68268a445d89aa2adaff3428348829a742bba00"} err="failed to get container status \"54e2fa9d516e4d2c8bbf0149a68268a445d89aa2adaff3428348829a742bba00\": rpc error: code = NotFound desc = could not find container \"54e2fa9d516e4d2c8bbf0149a68268a445d89aa2adaff3428348829a742bba00\": container with ID starting with 54e2fa9d516e4d2c8bbf0149a68268a445d89aa2adaff3428348829a742bba00 not found: ID does not exist" Oct 30 09:47:26.684611 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:26.683862 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-74f944f77-jfq98"] Oct 30 09:47:26.704776 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:26.704711 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-74f944f77-jfq98"] Oct 30 09:47:28.263711 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:28.263682 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d0da42-776d-4a4a-980d-c2494e623bc9" path="/var/lib/kubelet/pods/a6d0da42-776d-4a4a-980d-c2494e623bc9/volumes" Oct 30 09:47:28.840562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:28.840527 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9"] Oct 30 09:47:28.841620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:28.841581 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a6d0da42-776d-4a4a-980d-c2494e623bc9" containerName="payload-processing" Oct 30 09:47:28.841728 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:28.841621 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d0da42-776d-4a4a-980d-c2494e623bc9" containerName="payload-processing" Oct 30 09:47:28.841789 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:28.841778 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d0da42-776d-4a4a-980d-c2494e623bc9" containerName="payload-processing" Oct 30 09:47:28.847114 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:28.847096 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9" Oct 30 09:47:28.862924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:28.862900 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9"] Oct 30 09:47:28.871618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:28.871580 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:47:28.871780 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:28.871704 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:47:28.927826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:28.927796 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98skr\" (UniqueName: \"kubernetes.io/projected/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-kube-api-access-98skr\") pod \"maas-api-revoke-sub-a1324ac1-vkqt9\" (UID: \"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9" Oct 30 09:47:28.928048 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:28.928014 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1324ac1-vkqt9\" (UID: \"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9" Oct 30 09:47:28.928093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:28.928048 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1324ac1-vkqt9\" (UID: \"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9" Oct 30 09:47:29.029694 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:29.029661 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1324ac1-vkqt9\" (UID: \"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9" Oct 30 09:47:29.029865 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:29.029719 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1324ac1-vkqt9\" (UID: \"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9" Oct 30 09:47:29.029918 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:29.029871 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98skr\" (UniqueName: \"kubernetes.io/projected/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-kube-api-access-98skr\") pod \"maas-api-revoke-sub-a1324ac1-vkqt9\" (UID: \"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9" Oct 30 09:47:29.030474 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:29.030451 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1324ac1-vkqt9\" (UID: \"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9" Oct 30 09:47:29.032366 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:29.032349 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1324ac1-vkqt9\" (UID: \"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9" Oct 30 09:47:29.051627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:29.051567 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98skr\" (UniqueName: \"kubernetes.io/projected/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-kube-api-access-98skr\") pod \"maas-api-revoke-sub-a1324ac1-vkqt9\" (UID: \"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9" Oct 30 09:47:29.161827 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:29.161794 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9" Oct 30 09:47:29.469262 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:47:29.469218 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0b2a53_fdba_4dfd_a4e4_65acb7f4fe72.slice/crio-fb399da4a9a3db6c546c8c2cfb3b327a0ddb514078601c1b8903e767cfda7517 WatchSource:0}: Error finding container fb399da4a9a3db6c546c8c2cfb3b327a0ddb514078601c1b8903e767cfda7517: Status 404 returned error can't find the container with id fb399da4a9a3db6c546c8c2cfb3b327a0ddb514078601c1b8903e767cfda7517 Oct 30 09:47:29.469652 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:29.469459 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9"] Oct 30 09:47:29.657310 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:29.657284 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9" event={"ID":"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72","Type":"ContainerStarted","Data":"fb399da4a9a3db6c546c8c2cfb3b327a0ddb514078601c1b8903e767cfda7517"} Oct 30 09:47:30.666505 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:30.666473 5938 generic.go:356] "Generic (PLEG): container finished" podID="6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72" containerID="5bf60015c1f14270793ea54044a06de2f60010269267f27526337639cdd8a385" exitCode=0 Oct 30 09:47:30.666923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:30.666564 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9" event={"ID":"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72","Type":"ContainerDied","Data":"5bf60015c1f14270793ea54044a06de2f60010269267f27526337639cdd8a385"} Oct 30 09:47:31.819830 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:31.819803 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9" Oct 30 09:47:31.876936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:31.876901 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-kube-api-access-98skr\" (UniqueName: \"kubernetes.io/projected/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-kube-api-access-98skr\") pod \"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72\" (UID: \"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72\") " Oct 30 09:47:31.877642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:31.877032 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-maas-api-openshift-ca\") pod \"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72\" (UID: \"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72\") " Oct 30 09:47:31.877786 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:31.877700 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-maas-api-openshift-ca" pod "6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72" (UID: "6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:47:31.877786 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:31.877772 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-maas-api-xks-ca\") pod \"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72\" (UID: \"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72\") " Oct 30 09:47:31.878936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:31.878909 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:31.881110 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:31.881062 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-maas-api-xks-ca" pod "6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72" (UID: "6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:47:31.883797 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:31.883768 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-kube-api-access-98skr" pod "6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72" (UID: "6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72"). InnerVolumeSpecName "kube-api-access-98skr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:47:31.980309 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:31.980243 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98skr\" (UniqueName: \"kubernetes.io/projected/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-kube-api-access-98skr\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:31.980309 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:31.980271 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:32.256475 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:32.256387 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:47:32.683096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:32.683065 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9" event={"ID":"6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72","Type":"ContainerDied","Data":"fb399da4a9a3db6c546c8c2cfb3b327a0ddb514078601c1b8903e767cfda7517"} Oct 30 09:47:32.683096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:32.683086 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9" Oct 30 09:47:32.683320 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:32.683113 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb399da4a9a3db6c546c8c2cfb3b327a0ddb514078601c1b8903e767cfda7517" Oct 30 09:47:39.717639 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:39.717610 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d"] Oct 30 09:47:39.718283 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:39.718266 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72" containerName="revoke-keys" Oct 30 09:47:39.718283 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:39.718283 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72" containerName="revoke-keys" Oct 30 09:47:39.718432 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:39.718422 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72" containerName="revoke-keys" Oct 30 09:47:39.722383 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:39.722368 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" Oct 30 09:47:39.740285 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:39.740259 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d"] Oct 30 09:47:39.749398 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:39.749379 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:47:39.749562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:39.749386 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:47:39.878972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:39.878936 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e17ac2d2-e63e-4959-951d-d8bc30d14303-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a07fb049-2tf8d\" (UID: \"e17ac2d2-e63e-4959-951d-d8bc30d14303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" Oct 30 09:47:39.879196 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:39.879173 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e17ac2d2-e63e-4959-951d-d8bc30d14303-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a07fb049-2tf8d\" (UID: \"e17ac2d2-e63e-4959-951d-d8bc30d14303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" Oct 30 09:47:39.879270 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:39.879215 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d72kh\" (UniqueName: \"kubernetes.io/projected/e17ac2d2-e63e-4959-951d-d8bc30d14303-kube-api-access-d72kh\") pod \"maas-api-revoke-sub-a07fb049-2tf8d\" (UID: \"e17ac2d2-e63e-4959-951d-d8bc30d14303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" Oct 30 09:47:39.980635 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:39.980556 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e17ac2d2-e63e-4959-951d-d8bc30d14303-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a07fb049-2tf8d\" (UID: \"e17ac2d2-e63e-4959-951d-d8bc30d14303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" Oct 30 09:47:39.980635 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:39.980612 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d72kh\" (UniqueName: \"kubernetes.io/projected/e17ac2d2-e63e-4959-951d-d8bc30d14303-kube-api-access-d72kh\") pod \"maas-api-revoke-sub-a07fb049-2tf8d\" (UID: \"e17ac2d2-e63e-4959-951d-d8bc30d14303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" Oct 30 09:47:39.980807 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:39.980702 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e17ac2d2-e63e-4959-951d-d8bc30d14303-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a07fb049-2tf8d\" (UID: \"e17ac2d2-e63e-4959-951d-d8bc30d14303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" Oct 30 09:47:39.981266 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:39.981244 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e17ac2d2-e63e-4959-951d-d8bc30d14303-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a07fb049-2tf8d\" (UID: \"e17ac2d2-e63e-4959-951d-d8bc30d14303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" Oct 30 09:47:39.982978 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:39.982954 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e17ac2d2-e63e-4959-951d-d8bc30d14303-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a07fb049-2tf8d\" (UID: \"e17ac2d2-e63e-4959-951d-d8bc30d14303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" Oct 30 09:47:40.000868 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:40.000846 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d72kh\" (UniqueName: \"kubernetes.io/projected/e17ac2d2-e63e-4959-951d-d8bc30d14303-kube-api-access-d72kh\") pod \"maas-api-revoke-sub-a07fb049-2tf8d\" (UID: \"e17ac2d2-e63e-4959-951d-d8bc30d14303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" Oct 30 09:47:40.037433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:40.037411 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" Oct 30 09:47:40.340987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:40.340959 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d"] Oct 30 09:47:40.341515 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:47:40.341492 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17ac2d2_e63e_4959_951d_d8bc30d14303.slice/crio-fd6c5f0243f7a1a31766154300e1e372817f1c1b8b5f9481d24f988882884673 WatchSource:0}: Error finding container fd6c5f0243f7a1a31766154300e1e372817f1c1b8b5f9481d24f988882884673: Status 404 returned error can't find the container with id fd6c5f0243f7a1a31766154300e1e372817f1c1b8b5f9481d24f988882884673 Oct 30 09:47:40.743854 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:40.743824 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" event={"ID":"e17ac2d2-e63e-4959-951d-d8bc30d14303","Type":"ContainerStarted","Data":"a5a5c43c5bea3503c9ecd11ed6ee3f6f3bd3febba92c3f6c38e5d77b91bc1443"} Oct 30 09:47:40.744169 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:40.743878 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" event={"ID":"e17ac2d2-e63e-4959-951d-d8bc30d14303","Type":"ContainerStarted","Data":"fd6c5f0243f7a1a31766154300e1e372817f1c1b8b5f9481d24f988882884673"} Oct 30 09:47:40.785156 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:40.785105 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" podStartSLOduration=1.7850884630000001 podStartE2EDuration="1.785088463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:47:40.783397205 +0000 UTC m=+3178.646103026" watchObservedRunningTime="2026-10-30 09:47:40.785088463 +0000 UTC m=+3178.647794285" Oct 30 09:47:41.753379 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:41.753345 5938 generic.go:356] "Generic (PLEG): container finished" podID="e17ac2d2-e63e-4959-951d-d8bc30d14303" containerID="a5a5c43c5bea3503c9ecd11ed6ee3f6f3bd3febba92c3f6c38e5d77b91bc1443" exitCode=0 Oct 30 09:47:41.753878 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:41.753433 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" event={"ID":"e17ac2d2-e63e-4959-951d-d8bc30d14303","Type":"ContainerDied","Data":"a5a5c43c5bea3503c9ecd11ed6ee3f6f3bd3febba92c3f6c38e5d77b91bc1443"} Oct 30 09:47:42.257172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:42.257137 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:47:42.918054 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:42.918028 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" Oct 30 09:47:43.019987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:43.019914 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e17ac2d2-e63e-4959-951d-d8bc30d14303-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e17ac2d2-e63e-4959-951d-d8bc30d14303-maas-api-xks-ca\") pod \"e17ac2d2-e63e-4959-951d-d8bc30d14303\" (UID: \"e17ac2d2-e63e-4959-951d-d8bc30d14303\") " Oct 30 09:47:43.019987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:43.019985 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e17ac2d2-e63e-4959-951d-d8bc30d14303-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e17ac2d2-e63e-4959-951d-d8bc30d14303-maas-api-openshift-ca\") pod \"e17ac2d2-e63e-4959-951d-d8bc30d14303\" (UID: \"e17ac2d2-e63e-4959-951d-d8bc30d14303\") " Oct 30 09:47:43.020234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:43.020143 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17ac2d2-e63e-4959-951d-d8bc30d14303-kube-api-access-d72kh\" (UniqueName: \"kubernetes.io/projected/e17ac2d2-e63e-4959-951d-d8bc30d14303-kube-api-access-d72kh\") pod \"e17ac2d2-e63e-4959-951d-d8bc30d14303\" (UID: \"e17ac2d2-e63e-4959-951d-d8bc30d14303\") " Oct 30 09:47:43.020393 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:43.020369 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17ac2d2-e63e-4959-951d-d8bc30d14303-maas-api-openshift-ca" pod "e17ac2d2-e63e-4959-951d-d8bc30d14303" (UID: "e17ac2d2-e63e-4959-951d-d8bc30d14303"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:47:43.020636 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:43.020617 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e17ac2d2-e63e-4959-951d-d8bc30d14303-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:43.022199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:43.022176 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17ac2d2-e63e-4959-951d-d8bc30d14303-kube-api-access-d72kh" pod "e17ac2d2-e63e-4959-951d-d8bc30d14303" (UID: "e17ac2d2-e63e-4959-951d-d8bc30d14303"). InnerVolumeSpecName "kube-api-access-d72kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:47:43.022287 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:43.022247 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e17ac2d2-e63e-4959-951d-d8bc30d14303-maas-api-xks-ca" pod "e17ac2d2-e63e-4959-951d-d8bc30d14303" (UID: "e17ac2d2-e63e-4959-951d-d8bc30d14303"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:47:43.122259 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:43.122235 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d72kh\" (UniqueName: \"kubernetes.io/projected/e17ac2d2-e63e-4959-951d-d8bc30d14303-kube-api-access-d72kh\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:43.122259 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:43.122259 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e17ac2d2-e63e-4959-951d-d8bc30d14303-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:43.769601 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:43.769559 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" Oct 30 09:47:43.769788 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:43.769624 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d" event={"ID":"e17ac2d2-e63e-4959-951d-d8bc30d14303","Type":"ContainerDied","Data":"fd6c5f0243f7a1a31766154300e1e372817f1c1b8b5f9481d24f988882884673"} Oct 30 09:47:43.769788 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:43.769672 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd6c5f0243f7a1a31766154300e1e372817f1c1b8b5f9481d24f988882884673" Oct 30 09:47:44.002828 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:44.002794 5938 scope.go:122] "RemoveContainer" containerID="2df9bb9be9b554800311179f385b93e0063e763f778feaaeec0ffa4d08cca668" Oct 30 09:47:45.895792 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:47:45.895729 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:47:51.252301 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.252266 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h"] Oct 30 09:47:51.253057 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.253033 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e17ac2d2-e63e-4959-951d-d8bc30d14303" containerName="revoke-keys" Oct 30 09:47:51.253057 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.253057 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e17ac2d2-e63e-4959-951d-d8bc30d14303" containerName="revoke-keys" Oct 30 09:47:51.253254 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.253242 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e17ac2d2-e63e-4959-951d-d8bc30d14303" containerName="revoke-keys" Oct 30 09:47:51.258380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.258363 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:51.274548 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.274526 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h"] Oct 30 09:47:51.385757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.385719 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7rmb\" (UniqueName: \"kubernetes.io/projected/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-kube-api-access-t7rmb\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h\" (UID: \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:51.385757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.385759 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h\" (UID: \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:51.385977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.385799 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h\" (UID: \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:51.385977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.385896 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h\" (UID: \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:51.487192 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.487155 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7rmb\" (UniqueName: \"kubernetes.io/projected/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-kube-api-access-t7rmb\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h\" (UID: \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:51.487397 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.487205 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h\" (UID: \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:51.487397 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.487257 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h\" (UID: \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:51.487397 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.487312 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h\" (UID: \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:51.488066 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.488043 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h\" (UID: \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:51.488150 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.488115 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h\" (UID: \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:51.489796 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.489778 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h\" (UID: \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:51.508157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.508093 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7rmb\" (UniqueName: \"kubernetes.io/projected/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-kube-api-access-t7rmb\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h\" (UID: \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:51.572803 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.572781 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:51.880775 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.880745 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h"] Oct 30 09:47:51.954310 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.954279 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h"] Oct 30 09:47:51.958141 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:51.958118 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm"] Oct 30 09:47:52.256712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:52.256624 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:47:52.865693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:52.865657 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" event={"ID":"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290","Type":"ContainerStarted","Data":"6f61d1baeb55b03347345aa1f65769b49b15e8b2cd1cf78ca54755e01a5b12c6"} Oct 30 09:47:52.865851 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:52.865708 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" event={"ID":"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290","Type":"ContainerStarted","Data":"fd41f94e475bdf69efe1d5f48ebbac42054a638129d3e202b5b83095eaa2b731"} Oct 30 09:47:52.865851 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:52.865728 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:52.908017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:52.907970 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" podStartSLOduration=1.907957021 podStartE2EDuration="1.907957021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:47:52.90650572 +0000 UTC m=+3190.769211543" watchObservedRunningTime="2026-10-30 09:47:52.907957021 +0000 UTC m=+3190.770662842" Oct 30 09:47:54.157898 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.157867 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr"] Oct 30 09:47:54.162568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.162549 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr" Oct 30 09:47:54.181497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.181453 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr"] Oct 30 09:47:54.189650 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.189630 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:47:54.189769 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.189705 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:47:54.330570 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.330526 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j95h\" (UniqueName: \"kubernetes.io/projected/09e48367-fdd0-46b5-b288-be6c743ffa9a-kube-api-access-9j95h\") pod \"maas-api-revoke-sub-b4bb0e57-w2sjr\" (UID: \"09e48367-fdd0-46b5-b288-be6c743ffa9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr" Oct 30 09:47:54.330763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.330700 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e48367-fdd0-46b5-b288-be6c743ffa9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4bb0e57-w2sjr\" (UID: \"09e48367-fdd0-46b5-b288-be6c743ffa9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr" Oct 30 09:47:54.330926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.330907 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e48367-fdd0-46b5-b288-be6c743ffa9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4bb0e57-w2sjr\" (UID: \"09e48367-fdd0-46b5-b288-be6c743ffa9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr" Oct 30 09:47:54.432011 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.431939 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e48367-fdd0-46b5-b288-be6c743ffa9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4bb0e57-w2sjr\" (UID: \"09e48367-fdd0-46b5-b288-be6c743ffa9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr" Oct 30 09:47:54.432011 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.431988 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j95h\" (UniqueName: \"kubernetes.io/projected/09e48367-fdd0-46b5-b288-be6c743ffa9a-kube-api-access-9j95h\") pod \"maas-api-revoke-sub-b4bb0e57-w2sjr\" (UID: \"09e48367-fdd0-46b5-b288-be6c743ffa9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr" Oct 30 09:47:54.432206 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.432071 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e48367-fdd0-46b5-b288-be6c743ffa9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4bb0e57-w2sjr\" (UID: \"09e48367-fdd0-46b5-b288-be6c743ffa9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr" Oct 30 09:47:54.432709 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.432687 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e48367-fdd0-46b5-b288-be6c743ffa9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4bb0e57-w2sjr\" (UID: \"09e48367-fdd0-46b5-b288-be6c743ffa9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr" Oct 30 09:47:54.434434 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.434413 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e48367-fdd0-46b5-b288-be6c743ffa9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4bb0e57-w2sjr\" (UID: \"09e48367-fdd0-46b5-b288-be6c743ffa9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr" Oct 30 09:47:54.454095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.454067 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j95h\" (UniqueName: \"kubernetes.io/projected/09e48367-fdd0-46b5-b288-be6c743ffa9a-kube-api-access-9j95h\") pod \"maas-api-revoke-sub-b4bb0e57-w2sjr\" (UID: \"09e48367-fdd0-46b5-b288-be6c743ffa9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr" Oct 30 09:47:54.477555 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.477534 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr" Oct 30 09:47:54.780943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.780910 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr"] Oct 30 09:47:54.781322 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:47:54.781299 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e48367_fdd0_46b5_b288_be6c743ffa9a.slice/crio-e7acd2c76578cad75287b71e06f9645d9aaee1caedf7ab2da402f84cbce54263 WatchSource:0}: Error finding container e7acd2c76578cad75287b71e06f9645d9aaee1caedf7ab2da402f84cbce54263: Status 404 returned error can't find the container with id e7acd2c76578cad75287b71e06f9645d9aaee1caedf7ab2da402f84cbce54263 Oct 30 09:47:54.882723 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:54.882695 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr" event={"ID":"09e48367-fdd0-46b5-b288-be6c743ffa9a","Type":"ContainerStarted","Data":"e7acd2c76578cad75287b71e06f9645d9aaee1caedf7ab2da402f84cbce54263"} Oct 30 09:47:55.891890 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:55.891857 5938 generic.go:356] "Generic (PLEG): container finished" podID="09e48367-fdd0-46b5-b288-be6c743ffa9a" containerID="a1b8724c5a0af924c6ec358a678438cd72388bcd2d656b575f2ddb342be995e6" exitCode=0 Oct 30 09:47:55.892307 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:55.891944 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr" event={"ID":"09e48367-fdd0-46b5-b288-be6c743ffa9a","Type":"ContainerDied","Data":"a1b8724c5a0af924c6ec358a678438cd72388bcd2d656b575f2ddb342be995e6"} Oct 30 09:47:56.958743 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:56.958713 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" podUID="c0ad6b7a-5034-44da-94e4-d9e97192b749" containerName="payload-processing" containerID="cri-o://e844ddf279d34884ae37d7e3da5fca25084c590c4d88008a708ec928bce929fe" gracePeriod=25 Oct 30 09:47:57.097222 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.097186 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr" Oct 30 09:47:57.250693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.250669 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:47:57.278067 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.278042 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09e48367-fdd0-46b5-b288-be6c743ffa9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e48367-fdd0-46b5-b288-be6c743ffa9a-maas-api-xks-ca\") pod \"09e48367-fdd0-46b5-b288-be6c743ffa9a\" (UID: \"09e48367-fdd0-46b5-b288-be6c743ffa9a\") " Oct 30 09:47:57.278245 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.278230 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09e48367-fdd0-46b5-b288-be6c743ffa9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e48367-fdd0-46b5-b288-be6c743ffa9a-maas-api-openshift-ca\") pod \"09e48367-fdd0-46b5-b288-be6c743ffa9a\" (UID: \"09e48367-fdd0-46b5-b288-be6c743ffa9a\") " Oct 30 09:47:57.278402 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.278377 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09e48367-fdd0-46b5-b288-be6c743ffa9a-kube-api-access-9j95h\" (UniqueName: \"kubernetes.io/projected/09e48367-fdd0-46b5-b288-be6c743ffa9a-kube-api-access-9j95h\") pod \"09e48367-fdd0-46b5-b288-be6c743ffa9a\" (UID: \"09e48367-fdd0-46b5-b288-be6c743ffa9a\") " Oct 30 09:47:57.278568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.278543 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e48367-fdd0-46b5-b288-be6c743ffa9a-maas-api-openshift-ca" pod "09e48367-fdd0-46b5-b288-be6c743ffa9a" (UID: "09e48367-fdd0-46b5-b288-be6c743ffa9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:47:57.278865 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.278848 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e48367-fdd0-46b5-b288-be6c743ffa9a-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:57.280114 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.280092 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e48367-fdd0-46b5-b288-be6c743ffa9a-maas-api-xks-ca" pod "09e48367-fdd0-46b5-b288-be6c743ffa9a" (UID: "09e48367-fdd0-46b5-b288-be6c743ffa9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:47:57.280454 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.280438 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e48367-fdd0-46b5-b288-be6c743ffa9a-kube-api-access-9j95h" pod "09e48367-fdd0-46b5-b288-be6c743ffa9a" (UID: "09e48367-fdd0-46b5-b288-be6c743ffa9a"). InnerVolumeSpecName "kube-api-access-9j95h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:47:57.379945 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.379915 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ad6b7a-5034-44da-94e4-d9e97192b749-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0ad6b7a-5034-44da-94e4-d9e97192b749-provider-credentials\") pod \"c0ad6b7a-5034-44da-94e4-d9e97192b749\" (UID: \"c0ad6b7a-5034-44da-94e4-d9e97192b749\") " Oct 30 09:47:57.380114 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.379974 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ad6b7a-5034-44da-94e4-d9e97192b749-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ad6b7a-5034-44da-94e4-d9e97192b749-plugins-config-volume\") pod \"c0ad6b7a-5034-44da-94e4-d9e97192b749\" (UID: \"c0ad6b7a-5034-44da-94e4-d9e97192b749\") " Oct 30 09:47:57.380114 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.380064 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ad6b7a-5034-44da-94e4-d9e97192b749-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0ad6b7a-5034-44da-94e4-d9e97192b749-routing-overlay-volume\") pod \"c0ad6b7a-5034-44da-94e4-d9e97192b749\" (UID: \"c0ad6b7a-5034-44da-94e4-d9e97192b749\") " Oct 30 09:47:57.380204 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.380112 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ad6b7a-5034-44da-94e4-d9e97192b749-kube-api-access-s8stx\" (UniqueName: \"kubernetes.io/projected/c0ad6b7a-5034-44da-94e4-d9e97192b749-kube-api-access-s8stx\") pod \"c0ad6b7a-5034-44da-94e4-d9e97192b749\" (UID: \"c0ad6b7a-5034-44da-94e4-d9e97192b749\") " Oct 30 09:47:57.380388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.380360 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ad6b7a-5034-44da-94e4-d9e97192b749-plugins-config-volume" pod "c0ad6b7a-5034-44da-94e4-d9e97192b749" (UID: "c0ad6b7a-5034-44da-94e4-d9e97192b749"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:47:57.380535 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.380461 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ad6b7a-5034-44da-94e4-d9e97192b749-routing-overlay-volume" pod "c0ad6b7a-5034-44da-94e4-d9e97192b749" (UID: "c0ad6b7a-5034-44da-94e4-d9e97192b749"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:47:57.380859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.380841 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j95h\" (UniqueName: \"kubernetes.io/projected/09e48367-fdd0-46b5-b288-be6c743ffa9a-kube-api-access-9j95h\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:57.380936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.380860 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ad6b7a-5034-44da-94e4-d9e97192b749-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:57.380936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.380871 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e48367-fdd0-46b5-b288-be6c743ffa9a-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:57.380936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.380880 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0ad6b7a-5034-44da-94e4-d9e97192b749-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:57.382410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.382387 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ad6b7a-5034-44da-94e4-d9e97192b749-kube-api-access-s8stx" pod "c0ad6b7a-5034-44da-94e4-d9e97192b749" (UID: "c0ad6b7a-5034-44da-94e4-d9e97192b749"). InnerVolumeSpecName "kube-api-access-s8stx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:47:57.382542 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.382518 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ad6b7a-5034-44da-94e4-d9e97192b749-provider-credentials" pod "c0ad6b7a-5034-44da-94e4-d9e97192b749" (UID: "c0ad6b7a-5034-44da-94e4-d9e97192b749"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:47:57.481988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.481928 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0ad6b7a-5034-44da-94e4-d9e97192b749-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:57.481988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.481950 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8stx\" (UniqueName: \"kubernetes.io/projected/c0ad6b7a-5034-44da-94e4-d9e97192b749-kube-api-access-s8stx\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:57.866561 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.866519 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" podUID="c3d4beba-bbc4-4a27-b0ba-953c8ecfb290" containerName="payload-processing" containerID="cri-o://6f61d1baeb55b03347345aa1f65769b49b15e8b2cd1cf78ca54755e01a5b12c6" gracePeriod=25 Oct 30 09:47:57.908147 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.908122 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr" Oct 30 09:47:57.908276 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.908155 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr" event={"ID":"09e48367-fdd0-46b5-b288-be6c743ffa9a","Type":"ContainerDied","Data":"e7acd2c76578cad75287b71e06f9645d9aaee1caedf7ab2da402f84cbce54263"} Oct 30 09:47:57.908276 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.908202 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7acd2c76578cad75287b71e06f9645d9aaee1caedf7ab2da402f84cbce54263" Oct 30 09:47:57.909756 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.909731 5938 generic.go:356] "Generic (PLEG): container finished" podID="c0ad6b7a-5034-44da-94e4-d9e97192b749" containerID="e844ddf279d34884ae37d7e3da5fca25084c590c4d88008a708ec928bce929fe" exitCode=0 Oct 30 09:47:57.909850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.909776 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" Oct 30 09:47:57.909850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.909816 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" event={"ID":"c0ad6b7a-5034-44da-94e4-d9e97192b749","Type":"ContainerDied","Data":"e844ddf279d34884ae37d7e3da5fca25084c590c4d88008a708ec928bce929fe"} Oct 30 09:47:57.909920 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.909878 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm" event={"ID":"c0ad6b7a-5034-44da-94e4-d9e97192b749","Type":"ContainerDied","Data":"8b36c1ef0578d3cf16c6a73cae8b20353f1f986e4e16073f4e79114e0cd7c760"} Oct 30 09:47:57.909920 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.909911 5938 scope.go:122] "RemoveContainer" containerID="e844ddf279d34884ae37d7e3da5fca25084c590c4d88008a708ec928bce929fe" Oct 30 09:47:57.968542 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.968519 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm"] Oct 30 09:47:57.972385 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.972364 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-94ff66fzm"] Oct 30 09:47:57.981255 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.981236 5938 scope.go:122] "RemoveContainer" containerID="e844ddf279d34884ae37d7e3da5fca25084c590c4d88008a708ec928bce929fe" Oct 30 09:47:57.981651 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:47:57.981626 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e844ddf279d34884ae37d7e3da5fca25084c590c4d88008a708ec928bce929fe\": container with ID starting with e844ddf279d34884ae37d7e3da5fca25084c590c4d88008a708ec928bce929fe not found: ID does not exist" containerID="e844ddf279d34884ae37d7e3da5fca25084c590c4d88008a708ec928bce929fe" Oct 30 09:47:57.981754 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:57.981660 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e844ddf279d34884ae37d7e3da5fca25084c590c4d88008a708ec928bce929fe"} err="failed to get container status \"e844ddf279d34884ae37d7e3da5fca25084c590c4d88008a708ec928bce929fe\": rpc error: code = NotFound desc = could not find container \"e844ddf279d34884ae37d7e3da5fca25084c590c4d88008a708ec928bce929fe\": container with ID starting with e844ddf279d34884ae37d7e3da5fca25084c590c4d88008a708ec928bce929fe not found: ID does not exist" Oct 30 09:47:58.162146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.162124 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:58.260969 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.260942 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ad6b7a-5034-44da-94e4-d9e97192b749" path="/var/lib/kubelet/pods/c0ad6b7a-5034-44da-94e4-d9e97192b749/volumes" Oct 30 09:47:58.294099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.292484 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-plugins-config-volume\") pod \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\" (UID: \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\") " Oct 30 09:47:58.294099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.292646 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-provider-credentials\") pod \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\" (UID: \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\") " Oct 30 09:47:58.294099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.292715 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-routing-overlay-volume\") pod \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\" (UID: \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\") " Oct 30 09:47:58.294099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.292915 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-kube-api-access-t7rmb\" (UniqueName: \"kubernetes.io/projected/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-kube-api-access-t7rmb\") pod \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\" (UID: \"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290\") " Oct 30 09:47:58.297158 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.296948 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-kube-api-access-t7rmb" pod "c3d4beba-bbc4-4a27-b0ba-953c8ecfb290" (UID: "c3d4beba-bbc4-4a27-b0ba-953c8ecfb290"). InnerVolumeSpecName "kube-api-access-t7rmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:47:58.297753 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.297454 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-plugins-config-volume" pod "c3d4beba-bbc4-4a27-b0ba-953c8ecfb290" (UID: "c3d4beba-bbc4-4a27-b0ba-953c8ecfb290"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:47:58.300194 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.300000 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-provider-credentials" pod "c3d4beba-bbc4-4a27-b0ba-953c8ecfb290" (UID: "c3d4beba-bbc4-4a27-b0ba-953c8ecfb290"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:47:58.301310 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.300406 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-routing-overlay-volume" pod "c3d4beba-bbc4-4a27-b0ba-953c8ecfb290" (UID: "c3d4beba-bbc4-4a27-b0ba-953c8ecfb290"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:47:58.394800 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.394739 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:58.394800 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.394768 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7rmb\" (UniqueName: \"kubernetes.io/projected/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-kube-api-access-t7rmb\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:58.394800 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.394782 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:58.394800 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.394796 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:47:58.921139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.921106 5938 generic.go:356] "Generic (PLEG): container finished" podID="c3d4beba-bbc4-4a27-b0ba-953c8ecfb290" containerID="6f61d1baeb55b03347345aa1f65769b49b15e8b2cd1cf78ca54755e01a5b12c6" exitCode=0 Oct 30 09:47:58.921356 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.921168 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" Oct 30 09:47:58.921356 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.921178 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" event={"ID":"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290","Type":"ContainerDied","Data":"6f61d1baeb55b03347345aa1f65769b49b15e8b2cd1cf78ca54755e01a5b12c6"} Oct 30 09:47:58.921356 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.921257 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h" event={"ID":"c3d4beba-bbc4-4a27-b0ba-953c8ecfb290","Type":"ContainerDied","Data":"fd41f94e475bdf69efe1d5f48ebbac42054a638129d3e202b5b83095eaa2b731"} Oct 30 09:47:58.921356 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.921278 5938 scope.go:122] "RemoveContainer" containerID="6f61d1baeb55b03347345aa1f65769b49b15e8b2cd1cf78ca54755e01a5b12c6" Oct 30 09:47:58.932886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.932868 5938 scope.go:122] "RemoveContainer" containerID="6f61d1baeb55b03347345aa1f65769b49b15e8b2cd1cf78ca54755e01a5b12c6" Oct 30 09:47:58.933161 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:47:58.933142 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f61d1baeb55b03347345aa1f65769b49b15e8b2cd1cf78ca54755e01a5b12c6\": container with ID starting with 6f61d1baeb55b03347345aa1f65769b49b15e8b2cd1cf78ca54755e01a5b12c6 not found: ID does not exist" containerID="6f61d1baeb55b03347345aa1f65769b49b15e8b2cd1cf78ca54755e01a5b12c6" Oct 30 09:47:58.933236 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.933170 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f61d1baeb55b03347345aa1f65769b49b15e8b2cd1cf78ca54755e01a5b12c6"} err="failed to get container status \"6f61d1baeb55b03347345aa1f65769b49b15e8b2cd1cf78ca54755e01a5b12c6\": rpc error: code = NotFound desc = could not find container \"6f61d1baeb55b03347345aa1f65769b49b15e8b2cd1cf78ca54755e01a5b12c6\": container with ID starting with 6f61d1baeb55b03347345aa1f65769b49b15e8b2cd1cf78ca54755e01a5b12c6 not found: ID does not exist" Oct 30 09:47:58.977844 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.977824 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h"] Oct 30 09:47:58.981797 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:47:58.981776 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58ccqh88h"] Oct 30 09:48:00.258402 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:00.258368 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3d4beba-bbc4-4a27-b0ba-953c8ecfb290" path="/var/lib/kubelet/pods/c3d4beba-bbc4-4a27-b0ba-953c8ecfb290/volumes" Oct 30 09:48:02.257051 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:02.257011 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:48:05.031496 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:05.031464 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn"] Oct 30 09:48:05.044633 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:05.044604 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ec72d3-5zsvn"] Oct 30 09:48:05.457861 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:05.457830 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9"] Oct 30 09:48:05.459041 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:05.459011 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:48:06.262187 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:06.262151 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5563d9a-edff-4a2e-9dcd-001e876851f8" path="/var/lib/kubelet/pods/d5563d9a-edff-4a2e-9dcd-001e876851f8/volumes" Oct 30 09:48:07.183036 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.183008 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k"] Oct 30 09:48:07.183835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.183792 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c3d4beba-bbc4-4a27-b0ba-953c8ecfb290" containerName="payload-processing" Oct 30 09:48:07.183835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.183811 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d4beba-bbc4-4a27-b0ba-953c8ecfb290" containerName="payload-processing" Oct 30 09:48:07.184160 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.183853 5938 cpu_manager.go:414] "containerMap: removing container" podUID="09e48367-fdd0-46b5-b288-be6c743ffa9a" containerName="revoke-keys" Oct 30 09:48:07.184160 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.183861 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e48367-fdd0-46b5-b288-be6c743ffa9a" containerName="revoke-keys" Oct 30 09:48:07.184160 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.183886 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c0ad6b7a-5034-44da-94e4-d9e97192b749" containerName="payload-processing" Oct 30 09:48:07.184160 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.183893 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ad6b7a-5034-44da-94e4-d9e97192b749" containerName="payload-processing" Oct 30 09:48:07.184160 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.184058 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d4beba-bbc4-4a27-b0ba-953c8ecfb290" containerName="payload-processing" Oct 30 09:48:07.184160 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.184085 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e48367-fdd0-46b5-b288-be6c743ffa9a" containerName="revoke-keys" Oct 30 09:48:07.184160 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.184098 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ad6b7a-5034-44da-94e4-d9e97192b749" containerName="payload-processing" Oct 30 09:48:07.189747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.189727 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:07.206717 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.206692 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"agc-cred-6d0ba7c4e69e\"" Oct 30 09:48:07.209036 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.209013 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k"] Oct 30 09:48:07.215664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.215629 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"payload-processing-external-model-e2e--185456a2-dockercfg-hwnjt\"" Oct 30 09:48:07.336208 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.336173 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbk68\" (UniqueName: \"kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-kube-api-access-hbk68\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k\" (UID: \"8f797687-88d5-415a-8f8c-176acb3fd17a\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:07.336726 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.336402 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k\" (UID: \"8f797687-88d5-415a-8f8c-176acb3fd17a\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:07.336806 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.336716 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f797687-88d5-415a-8f8c-176acb3fd17a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k\" (UID: \"8f797687-88d5-415a-8f8c-176acb3fd17a\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:07.336806 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.336755 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f797687-88d5-415a-8f8c-176acb3fd17a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k\" (UID: \"8f797687-88d5-415a-8f8c-176acb3fd17a\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:07.443657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.438561 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f797687-88d5-415a-8f8c-176acb3fd17a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k\" (UID: \"8f797687-88d5-415a-8f8c-176acb3fd17a\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:07.443657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.438639 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f797687-88d5-415a-8f8c-176acb3fd17a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k\" (UID: \"8f797687-88d5-415a-8f8c-176acb3fd17a\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:07.443657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.438790 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbk68\" (UniqueName: \"kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-kube-api-access-hbk68\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k\" (UID: \"8f797687-88d5-415a-8f8c-176acb3fd17a\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:07.443657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.438918 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k\" (UID: \"8f797687-88d5-415a-8f8c-176acb3fd17a\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:07.449662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.446295 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f797687-88d5-415a-8f8c-176acb3fd17a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k\" (UID: \"8f797687-88d5-415a-8f8c-176acb3fd17a\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:07.449662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.446770 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f797687-88d5-415a-8f8c-176acb3fd17a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k\" (UID: \"8f797687-88d5-415a-8f8c-176acb3fd17a\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:07.449662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.448107 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k\" (UID: \"8f797687-88d5-415a-8f8c-176acb3fd17a\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:07.489958 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.489926 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbk68\" (UniqueName: \"kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-kube-api-access-hbk68\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k\" (UID: \"8f797687-88d5-415a-8f8c-176acb3fd17a\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:07.507813 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.507785 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:07.810885 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.810852 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k"] Oct 30 09:48:07.812288 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:48:07.812256 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f797687_88d5_415a_8f8c_176acb3fd17a.slice/crio-70533f0d237bf482597171ce2d171c30182351f0619919a6ab453c1e82492b69 WatchSource:0}: Error finding container 70533f0d237bf482597171ce2d171c30182351f0619919a6ab453c1e82492b69: Status 404 returned error can't find the container with id 70533f0d237bf482597171ce2d171c30182351f0619919a6ab453c1e82492b69 Oct 30 09:48:07.920859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.920824 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q"] Oct 30 09:48:07.925557 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.925538 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:07.941272 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.941250 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-047b9f328507\"" Oct 30 09:48:07.943849 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.943827 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q"] Oct 30 09:48:07.999571 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.999535 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" event={"ID":"8f797687-88d5-415a-8f8c-176acb3fd17a","Type":"ContainerStarted","Data":"cebdf00e3c64c1b18d4f769f40ee9743b8a4b2630b5ff52911b5cf16af5b2b07"} Oct 30 09:48:07.999754 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:07.999636 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" event={"ID":"8f797687-88d5-415a-8f8c-176acb3fd17a","Type":"ContainerStarted","Data":"70533f0d237bf482597171ce2d171c30182351f0619919a6ab453c1e82492b69"} Oct 30 09:48:08.001222 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:08.001199 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:08.048305 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:08.048283 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-provider-credentials\") pod \"payload-processing-external-model-59686ffd74-bpf8q\" (UID: \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:08.048430 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:08.048413 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-routing-overlay-volume\") pod \"payload-processing-external-model-59686ffd74-bpf8q\" (UID: \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:08.048577 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:08.048561 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj848\" (UniqueName: \"kubernetes.io/projected/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-kube-api-access-cj848\") pod \"payload-processing-external-model-59686ffd74-bpf8q\" (UID: \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:08.048702 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:08.048686 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-plugins-config-volume\") pod \"payload-processing-external-model-59686ffd74-bpf8q\" (UID: \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:08.058409 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:08.058365 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" podStartSLOduration=1.058353008 podStartE2EDuration="1.058353008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:48:08.056471664 +0000 UTC m=+3205.919177485" watchObservedRunningTime="2026-10-30 09:48:08.058353008 +0000 UTC m=+3205.921058828" Oct 30 09:48:08.150036 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:08.150008 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-plugins-config-volume\") pod \"payload-processing-external-model-59686ffd74-bpf8q\" (UID: \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:08.150165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:08.150059 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-provider-credentials\") pod \"payload-processing-external-model-59686ffd74-bpf8q\" (UID: \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:08.150165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:08.150149 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-routing-overlay-volume\") pod \"payload-processing-external-model-59686ffd74-bpf8q\" (UID: \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:08.150308 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:08.150291 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj848\" (UniqueName: \"kubernetes.io/projected/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-kube-api-access-cj848\") pod \"payload-processing-external-model-59686ffd74-bpf8q\" (UID: \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:08.150912 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:08.150852 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-routing-overlay-volume\") pod \"payload-processing-external-model-59686ffd74-bpf8q\" (UID: \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:08.150912 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:08.150872 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-plugins-config-volume\") pod \"payload-processing-external-model-59686ffd74-bpf8q\" (UID: \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:08.152851 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:08.152831 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-provider-credentials\") pod \"payload-processing-external-model-59686ffd74-bpf8q\" (UID: \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:08.171026 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:08.171005 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj848\" (UniqueName: \"kubernetes.io/projected/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-kube-api-access-cj848\") pod \"payload-processing-external-model-59686ffd74-bpf8q\" (UID: \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:08.239859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:08.239835 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:08.548148 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:08.548119 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q"] Oct 30 09:48:08.548569 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:48:08.548501 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf41c33_4e96_4a8e_a050_f7449a5c66d6.slice/crio-297cebcd3376496a35dbe79e98652f7fb576edf8df542bf7a9f81e9a08607013 WatchSource:0}: Error finding container 297cebcd3376496a35dbe79e98652f7fb576edf8df542bf7a9f81e9a08607013: Status 404 returned error can't find the container with id 297cebcd3376496a35dbe79e98652f7fb576edf8df542bf7a9f81e9a08607013 Oct 30 09:48:09.012092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:09.012055 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" event={"ID":"1bf41c33-4e96-4a8e-a050-f7449a5c66d6","Type":"ContainerStarted","Data":"af382223d6535de89dacbd34c69bebaa9ef242977a605f187fcb1edae18810c7"} Oct 30 09:48:09.012289 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:09.012115 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" event={"ID":"1bf41c33-4e96-4a8e-a050-f7449a5c66d6","Type":"ContainerStarted","Data":"297cebcd3376496a35dbe79e98652f7fb576edf8df542bf7a9f81e9a08607013"} Oct 30 09:48:09.012364 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:09.012311 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:09.058260 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:09.058201 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" podStartSLOduration=2.058180429 podStartE2EDuration="2.058180429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:48:09.05246221 +0000 UTC m=+3206.915168032" watchObservedRunningTime="2026-10-30 09:48:09.058180429 +0000 UTC m=+3206.920886251" Oct 30 09:48:09.169899 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:09.169868 5938 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88cbc7/agc-cred-6d0ba7c4e69e: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:48:09.169899 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:09.169896 5938 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:48:09.170184 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:09.169955 5938 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-provider-credentials podName:8f797687-88d5-415a-8f8c-176acb3fd17a nodeName:}" failed. No retries permitted until 2026-10-30 09:48:09.669935083 +0000 UTC m=+3207.532640896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" (UID: "8f797687-88d5-415a-8f8c-176acb3fd17a") : secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:48:09.675415 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:09.675382 5938 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88cbc7/agc-cred-6d0ba7c4e69e: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:48:09.675415 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:09.675407 5938 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:48:09.675952 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:09.675465 5938 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-provider-credentials podName:8f797687-88d5-415a-8f8c-176acb3fd17a nodeName:}" failed. No retries permitted until 2026-10-30 09:48:10.675450982 +0000 UTC m=+3208.538156781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" (UID: "8f797687-88d5-415a-8f8c-176acb3fd17a") : secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:48:10.342056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:10.342021 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k"] Oct 30 09:48:10.458237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:10.458197 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" containerID="cri-o://880a65eeb1996d4b9c92ce68b1cbdd3b0e41623658132652e9747621f796cd85" gracePeriod=25 Oct 30 09:48:10.736630 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:10.735341 5938 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88cbc7/agc-cred-6d0ba7c4e69e: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:48:10.736630 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:10.735374 5938 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:48:10.736630 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:10.735441 5938 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-provider-credentials podName:8f797687-88d5-415a-8f8c-176acb3fd17a nodeName:}" failed. No retries permitted until 2026-10-30 09:48:12.73542074 +0000 UTC m=+3210.598126539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" (UID: "8f797687-88d5-415a-8f8c-176acb3fd17a") : secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:48:10.855928 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:10.855901 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:48:10.937902 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:10.937822 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d0be4a7-7a63-495e-a4df-560de711b86d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0be4a7-7a63-495e-a4df-560de711b86d-plugins-config-volume\") pod \"7d0be4a7-7a63-495e-a4df-560de711b86d\" (UID: \"7d0be4a7-7a63-495e-a4df-560de711b86d\") " Oct 30 09:48:10.938071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:10.937912 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0be4a7-7a63-495e-a4df-560de711b86d-kube-api-access-2rj7g\" (UniqueName: \"kubernetes.io/projected/7d0be4a7-7a63-495e-a4df-560de711b86d-kube-api-access-2rj7g\") pod \"7d0be4a7-7a63-495e-a4df-560de711b86d\" (UID: \"7d0be4a7-7a63-495e-a4df-560de711b86d\") " Oct 30 09:48:10.938071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:10.937946 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d0be4a7-7a63-495e-a4df-560de711b86d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d0be4a7-7a63-495e-a4df-560de711b86d-routing-overlay-volume\") pod \"7d0be4a7-7a63-495e-a4df-560de711b86d\" (UID: \"7d0be4a7-7a63-495e-a4df-560de711b86d\") " Oct 30 09:48:10.938071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:10.937968 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0be4a7-7a63-495e-a4df-560de711b86d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d0be4a7-7a63-495e-a4df-560de711b86d-provider-credentials\") pod \"7d0be4a7-7a63-495e-a4df-560de711b86d\" (UID: \"7d0be4a7-7a63-495e-a4df-560de711b86d\") " Oct 30 09:48:10.938342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:10.938315 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0be4a7-7a63-495e-a4df-560de711b86d-plugins-config-volume" pod "7d0be4a7-7a63-495e-a4df-560de711b86d" (UID: "7d0be4a7-7a63-495e-a4df-560de711b86d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:48:10.938671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:10.938639 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0be4a7-7a63-495e-a4df-560de711b86d-routing-overlay-volume" pod "7d0be4a7-7a63-495e-a4df-560de711b86d" (UID: "7d0be4a7-7a63-495e-a4df-560de711b86d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:48:10.938814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:10.938789 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0be4a7-7a63-495e-a4df-560de711b86d-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:10.938895 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:10.938823 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d0be4a7-7a63-495e-a4df-560de711b86d-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:10.940619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:10.940578 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0be4a7-7a63-495e-a4df-560de711b86d-provider-credentials" pod "7d0be4a7-7a63-495e-a4df-560de711b86d" (UID: "7d0be4a7-7a63-495e-a4df-560de711b86d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:48:10.940809 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:10.940790 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0be4a7-7a63-495e-a4df-560de711b86d-kube-api-access-2rj7g" pod "7d0be4a7-7a63-495e-a4df-560de711b86d" (UID: "7d0be4a7-7a63-495e-a4df-560de711b86d"). InnerVolumeSpecName "kube-api-access-2rj7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:48:11.028954 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:11.028926 5938 generic.go:356] "Generic (PLEG): container finished" podID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerID="880a65eeb1996d4b9c92ce68b1cbdd3b0e41623658132652e9747621f796cd85" exitCode=0 Oct 30 09:48:11.029106 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:11.029038 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" Oct 30 09:48:11.029106 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:11.029078 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" event={"ID":"7d0be4a7-7a63-495e-a4df-560de711b86d","Type":"ContainerDied","Data":"880a65eeb1996d4b9c92ce68b1cbdd3b0e41623658132652e9747621f796cd85"} Oct 30 09:48:11.029193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:11.029170 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9" event={"ID":"7d0be4a7-7a63-495e-a4df-560de711b86d","Type":"ContainerDied","Data":"39e57b65acbb92cef27744c7f18e20df52a4a1bb3d477172c66c0d5291b3a87b"} Oct 30 09:48:11.029246 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:11.029203 5938 scope.go:122] "RemoveContainer" containerID="880a65eeb1996d4b9c92ce68b1cbdd3b0e41623658132652e9747621f796cd85" Oct 30 09:48:11.040806 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:11.040786 5938 scope.go:122] "RemoveContainer" containerID="880a65eeb1996d4b9c92ce68b1cbdd3b0e41623658132652e9747621f796cd85" Oct 30 09:48:11.040943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:11.040925 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rj7g\" (UniqueName: \"kubernetes.io/projected/7d0be4a7-7a63-495e-a4df-560de711b86d-kube-api-access-2rj7g\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:11.040980 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:11.040951 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d0be4a7-7a63-495e-a4df-560de711b86d-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:11.041107 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:11.041090 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880a65eeb1996d4b9c92ce68b1cbdd3b0e41623658132652e9747621f796cd85\": container with ID starting with 880a65eeb1996d4b9c92ce68b1cbdd3b0e41623658132652e9747621f796cd85 not found: ID does not exist" containerID="880a65eeb1996d4b9c92ce68b1cbdd3b0e41623658132652e9747621f796cd85" Oct 30 09:48:11.041143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:11.041116 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880a65eeb1996d4b9c92ce68b1cbdd3b0e41623658132652e9747621f796cd85"} err="failed to get container status \"880a65eeb1996d4b9c92ce68b1cbdd3b0e41623658132652e9747621f796cd85\": rpc error: code = NotFound desc = could not find container \"880a65eeb1996d4b9c92ce68b1cbdd3b0e41623658132652e9747621f796cd85\": container with ID starting with 880a65eeb1996d4b9c92ce68b1cbdd3b0e41623658132652e9747621f796cd85 not found: ID does not exist" Oct 30 09:48:11.126023 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:11.125993 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9"] Oct 30 09:48:11.129797 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:11.129774 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-5df6tbfk9"] Oct 30 09:48:12.260294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:12.260260 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" path="/var/lib/kubelet/pods/7d0be4a7-7a63-495e-a4df-560de711b86d/volumes" Oct 30 09:48:12.646618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:12.644111 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q"] Oct 30 09:48:12.765305 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:12.765275 5938 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88cbc7/agc-cred-6d0ba7c4e69e: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:48:12.765305 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:12.765299 5938 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:48:12.765506 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:12.765360 5938 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-provider-credentials podName:8f797687-88d5-415a-8f8c-176acb3fd17a nodeName:}" failed. No retries permitted until 2026-10-30 09:48:16.765345902 +0000 UTC m=+3214.628051704 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" (UID: "8f797687-88d5-415a-8f8c-176acb3fd17a") : secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:48:15.808868 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:15.808794 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:48:16.029640 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:16.029602 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" podUID="8f797687-88d5-415a-8f8c-176acb3fd17a" containerName="payload-processing" containerID="cri-o://cebdf00e3c64c1b18d4f769f40ee9743b8a4b2630b5ff52911b5cf16af5b2b07" gracePeriod=25 Oct 30 09:48:16.311227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:16.311202 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:16.423495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:16.423419 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-provider-credentials\") pod \"8f797687-88d5-415a-8f8c-176acb3fd17a\" (UID: \"8f797687-88d5-415a-8f8c-176acb3fd17a\") " Oct 30 09:48:16.423847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:16.423695 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f797687-88d5-415a-8f8c-176acb3fd17a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f797687-88d5-415a-8f8c-176acb3fd17a-plugins-config-volume\") pod \"8f797687-88d5-415a-8f8c-176acb3fd17a\" (UID: \"8f797687-88d5-415a-8f8c-176acb3fd17a\") " Oct 30 09:48:16.423847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:16.423795 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-kube-api-access-hbk68\" (UniqueName: \"kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-kube-api-access-hbk68\") pod \"8f797687-88d5-415a-8f8c-176acb3fd17a\" (UID: \"8f797687-88d5-415a-8f8c-176acb3fd17a\") " Oct 30 09:48:16.423847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:16.423831 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f797687-88d5-415a-8f8c-176acb3fd17a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f797687-88d5-415a-8f8c-176acb3fd17a-routing-overlay-volume\") pod \"8f797687-88d5-415a-8f8c-176acb3fd17a\" (UID: \"8f797687-88d5-415a-8f8c-176acb3fd17a\") " Oct 30 09:48:16.424069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:16.424007 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f797687-88d5-415a-8f8c-176acb3fd17a-plugins-config-volume" pod "8f797687-88d5-415a-8f8c-176acb3fd17a" (UID: "8f797687-88d5-415a-8f8c-176acb3fd17a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:48:16.424231 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:16.424210 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f797687-88d5-415a-8f8c-176acb3fd17a-routing-overlay-volume" pod "8f797687-88d5-415a-8f8c-176acb3fd17a" (UID: "8f797687-88d5-415a-8f8c-176acb3fd17a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:48:16.424464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:16.424447 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f797687-88d5-415a-8f8c-176acb3fd17a-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:16.424503 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:16.424468 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f797687-88d5-415a-8f8c-176acb3fd17a-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:16.425939 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:16.425914 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-provider-credentials" pod "8f797687-88d5-415a-8f8c-176acb3fd17a" (UID: "8f797687-88d5-415a-8f8c-176acb3fd17a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:48:16.426094 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:16.426068 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-kube-api-access-hbk68" pod "8f797687-88d5-415a-8f8c-176acb3fd17a" (UID: "8f797687-88d5-415a-8f8c-176acb3fd17a"). InnerVolumeSpecName "kube-api-access-hbk68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:48:16.525437 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:16.525393 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:16.525437 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:16.525429 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbk68\" (UniqueName: \"kubernetes.io/projected/8f797687-88d5-415a-8f8c-176acb3fd17a-kube-api-access-hbk68\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:17.101516 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:17.101480 5938 generic.go:356] "Generic (PLEG): container finished" podID="8f797687-88d5-415a-8f8c-176acb3fd17a" containerID="cebdf00e3c64c1b18d4f769f40ee9743b8a4b2630b5ff52911b5cf16af5b2b07" exitCode=0 Oct 30 09:48:17.102059 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:17.101526 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" Oct 30 09:48:17.102059 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:17.101544 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" event={"ID":"8f797687-88d5-415a-8f8c-176acb3fd17a","Type":"ContainerDied","Data":"cebdf00e3c64c1b18d4f769f40ee9743b8a4b2630b5ff52911b5cf16af5b2b07"} Oct 30 09:48:17.102059 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:17.101616 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k" event={"ID":"8f797687-88d5-415a-8f8c-176acb3fd17a","Type":"ContainerDied","Data":"70533f0d237bf482597171ce2d171c30182351f0619919a6ab453c1e82492b69"} Oct 30 09:48:17.102059 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:17.101639 5938 scope.go:122] "RemoveContainer" containerID="cebdf00e3c64c1b18d4f769f40ee9743b8a4b2630b5ff52911b5cf16af5b2b07" Oct 30 09:48:17.113870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:17.113849 5938 scope.go:122] "RemoveContainer" containerID="cebdf00e3c64c1b18d4f769f40ee9743b8a4b2630b5ff52911b5cf16af5b2b07" Oct 30 09:48:17.114181 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:17.114156 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cebdf00e3c64c1b18d4f769f40ee9743b8a4b2630b5ff52911b5cf16af5b2b07\": container with ID starting with cebdf00e3c64c1b18d4f769f40ee9743b8a4b2630b5ff52911b5cf16af5b2b07 not found: ID does not exist" containerID="cebdf00e3c64c1b18d4f769f40ee9743b8a4b2630b5ff52911b5cf16af5b2b07" Oct 30 09:48:17.114318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:17.114190 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cebdf00e3c64c1b18d4f769f40ee9743b8a4b2630b5ff52911b5cf16af5b2b07"} err="failed to get container status \"cebdf00e3c64c1b18d4f769f40ee9743b8a4b2630b5ff52911b5cf16af5b2b07\": rpc error: code = NotFound desc = could not find container \"cebdf00e3c64c1b18d4f769f40ee9743b8a4b2630b5ff52911b5cf16af5b2b07\": container with ID starting with cebdf00e3c64c1b18d4f769f40ee9743b8a4b2630b5ff52911b5cf16af5b2b07 not found: ID does not exist" Oct 30 09:48:17.157620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:17.157574 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k"] Oct 30 09:48:17.170955 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:17.170933 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-65794jf7k"] Oct 30 09:48:17.644897 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:17.644856 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" podUID="1bf41c33-4e96-4a8e-a050-f7449a5c66d6" containerName="payload-processing" containerID="cri-o://af382223d6535de89dacbd34c69bebaa9ef242977a605f187fcb1edae18810c7" gracePeriod=25 Oct 30 09:48:17.925339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:17.925313 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:18.044702 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.044669 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-routing-overlay-volume\") pod \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\" (UID: \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\") " Oct 30 09:48:18.044702 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.044703 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-kube-api-access-cj848\" (UniqueName: \"kubernetes.io/projected/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-kube-api-access-cj848\") pod \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\" (UID: \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\") " Oct 30 09:48:18.044909 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.044800 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-provider-credentials\") pod \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\" (UID: \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\") " Oct 30 09:48:18.045079 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.045058 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-plugins-config-volume\") pod \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\" (UID: \"1bf41c33-4e96-4a8e-a050-f7449a5c66d6\") " Oct 30 09:48:18.045164 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.045090 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-routing-overlay-volume" pod "1bf41c33-4e96-4a8e-a050-f7449a5c66d6" (UID: "1bf41c33-4e96-4a8e-a050-f7449a5c66d6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:48:18.045463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.045436 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-plugins-config-volume" pod "1bf41c33-4e96-4a8e-a050-f7449a5c66d6" (UID: "1bf41c33-4e96-4a8e-a050-f7449a5c66d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:48:18.045654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.045640 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:18.045713 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.045657 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:18.047080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.047056 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-kube-api-access-cj848" pod "1bf41c33-4e96-4a8e-a050-f7449a5c66d6" (UID: "1bf41c33-4e96-4a8e-a050-f7449a5c66d6"). InnerVolumeSpecName "kube-api-access-cj848". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:48:18.047186 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.047135 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-provider-credentials" pod "1bf41c33-4e96-4a8e-a050-f7449a5c66d6" (UID: "1bf41c33-4e96-4a8e-a050-f7449a5c66d6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:48:18.110337 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.110305 5938 generic.go:356] "Generic (PLEG): container finished" podID="1bf41c33-4e96-4a8e-a050-f7449a5c66d6" containerID="af382223d6535de89dacbd34c69bebaa9ef242977a605f187fcb1edae18810c7" exitCode=0 Oct 30 09:48:18.110782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.110354 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" Oct 30 09:48:18.110782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.110385 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" event={"ID":"1bf41c33-4e96-4a8e-a050-f7449a5c66d6","Type":"ContainerDied","Data":"af382223d6535de89dacbd34c69bebaa9ef242977a605f187fcb1edae18810c7"} Oct 30 09:48:18.110782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.110448 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q" event={"ID":"1bf41c33-4e96-4a8e-a050-f7449a5c66d6","Type":"ContainerDied","Data":"297cebcd3376496a35dbe79e98652f7fb576edf8df542bf7a9f81e9a08607013"} Oct 30 09:48:18.110782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.110476 5938 scope.go:122] "RemoveContainer" containerID="af382223d6535de89dacbd34c69bebaa9ef242977a605f187fcb1edae18810c7" Oct 30 09:48:18.122482 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.122467 5938 scope.go:122] "RemoveContainer" containerID="af382223d6535de89dacbd34c69bebaa9ef242977a605f187fcb1edae18810c7" Oct 30 09:48:18.122771 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:18.122751 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af382223d6535de89dacbd34c69bebaa9ef242977a605f187fcb1edae18810c7\": container with ID starting with af382223d6535de89dacbd34c69bebaa9ef242977a605f187fcb1edae18810c7 not found: ID does not exist" containerID="af382223d6535de89dacbd34c69bebaa9ef242977a605f187fcb1edae18810c7" Oct 30 09:48:18.122854 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.122777 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af382223d6535de89dacbd34c69bebaa9ef242977a605f187fcb1edae18810c7"} err="failed to get container status \"af382223d6535de89dacbd34c69bebaa9ef242977a605f187fcb1edae18810c7\": rpc error: code = NotFound desc = could not find container \"af382223d6535de89dacbd34c69bebaa9ef242977a605f187fcb1edae18810c7\": container with ID starting with af382223d6535de89dacbd34c69bebaa9ef242977a605f187fcb1edae18810c7 not found: ID does not exist" Oct 30 09:48:18.147515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.147491 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:18.147515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.147514 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj848\" (UniqueName: \"kubernetes.io/projected/1bf41c33-4e96-4a8e-a050-f7449a5c66d6-kube-api-access-cj848\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:18.166735 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.166675 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q"] Oct 30 09:48:18.169875 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.169855 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-59686ffd74-bpf8q"] Oct 30 09:48:18.260071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.260044 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf41c33-4e96-4a8e-a050-f7449a5c66d6" path="/var/lib/kubelet/pods/1bf41c33-4e96-4a8e-a050-f7449a5c66d6/volumes" Oct 30 09:48:18.260532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:18.260514 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f797687-88d5-415a-8f8c-176acb3fd17a" path="/var/lib/kubelet/pods/8f797687-88d5-415a-8f8c-176acb3fd17a/volumes" Oct 30 09:48:26.027628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.027570 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x"] Oct 30 09:48:26.040601 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.040562 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab755ac3-zzv6x"] Oct 30 09:48:26.136254 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.136224 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn"] Oct 30 09:48:26.136923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.136908 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" Oct 30 09:48:26.136923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.136923 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" Oct 30 09:48:26.137053 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.136939 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1bf41c33-4e96-4a8e-a050-f7449a5c66d6" containerName="payload-processing" Oct 30 09:48:26.137053 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.136944 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf41c33-4e96-4a8e-a050-f7449a5c66d6" containerName="payload-processing" Oct 30 09:48:26.137053 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.136976 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8f797687-88d5-415a-8f8c-176acb3fd17a" containerName="payload-processing" Oct 30 09:48:26.137053 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.136982 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f797687-88d5-415a-8f8c-176acb3fd17a" containerName="payload-processing" Oct 30 09:48:26.137180 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.137096 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f797687-88d5-415a-8f8c-176acb3fd17a" containerName="payload-processing" Oct 30 09:48:26.137180 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.137109 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d0be4a7-7a63-495e-a4df-560de711b86d" containerName="payload-processing" Oct 30 09:48:26.137180 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.137116 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf41c33-4e96-4a8e-a050-f7449a5c66d6" containerName="payload-processing" Oct 30 09:48:26.142228 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.142208 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:26.158240 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.158216 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9dbfdb0617f2\"" Oct 30 09:48:26.159626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.159609 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn"] Oct 30 09:48:26.258090 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.258058 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fefff5c-b885-468b-bada-a1de738ab257" path="/var/lib/kubelet/pods/7fefff5c-b885-468b-bada-a1de738ab257/volumes" Oct 30 09:48:26.281370 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.281305 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67461890-6769-447f-af69-71f8d510e10c-routing-overlay-volume\") pod \"payload-processing-external-model-7859b55bb7-k5djn\" (UID: \"67461890-6769-447f-af69-71f8d510e10c\") " pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:26.281531 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.281509 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67461890-6769-447f-af69-71f8d510e10c-plugins-config-volume\") pod \"payload-processing-external-model-7859b55bb7-k5djn\" (UID: \"67461890-6769-447f-af69-71f8d510e10c\") " pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:26.281660 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.281644 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67461890-6769-447f-af69-71f8d510e10c-provider-credentials\") pod \"payload-processing-external-model-7859b55bb7-k5djn\" (UID: \"67461890-6769-447f-af69-71f8d510e10c\") " pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:26.281715 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.281681 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6mz4\" (UniqueName: \"kubernetes.io/projected/67461890-6769-447f-af69-71f8d510e10c-kube-api-access-j6mz4\") pod \"payload-processing-external-model-7859b55bb7-k5djn\" (UID: \"67461890-6769-447f-af69-71f8d510e10c\") " pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:26.382896 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.382864 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67461890-6769-447f-af69-71f8d510e10c-routing-overlay-volume\") pod \"payload-processing-external-model-7859b55bb7-k5djn\" (UID: \"67461890-6769-447f-af69-71f8d510e10c\") " pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:26.383057 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.383011 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67461890-6769-447f-af69-71f8d510e10c-plugins-config-volume\") pod \"payload-processing-external-model-7859b55bb7-k5djn\" (UID: \"67461890-6769-447f-af69-71f8d510e10c\") " pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:26.383116 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.383089 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67461890-6769-447f-af69-71f8d510e10c-provider-credentials\") pod \"payload-processing-external-model-7859b55bb7-k5djn\" (UID: \"67461890-6769-447f-af69-71f8d510e10c\") " pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:26.383192 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.383119 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6mz4\" (UniqueName: \"kubernetes.io/projected/67461890-6769-447f-af69-71f8d510e10c-kube-api-access-j6mz4\") pod \"payload-processing-external-model-7859b55bb7-k5djn\" (UID: \"67461890-6769-447f-af69-71f8d510e10c\") " pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:26.383742 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.383696 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67461890-6769-447f-af69-71f8d510e10c-routing-overlay-volume\") pod \"payload-processing-external-model-7859b55bb7-k5djn\" (UID: \"67461890-6769-447f-af69-71f8d510e10c\") " pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:26.384069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.383902 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67461890-6769-447f-af69-71f8d510e10c-plugins-config-volume\") pod \"payload-processing-external-model-7859b55bb7-k5djn\" (UID: \"67461890-6769-447f-af69-71f8d510e10c\") " pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:26.386336 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.386310 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67461890-6769-447f-af69-71f8d510e10c-provider-credentials\") pod \"payload-processing-external-model-7859b55bb7-k5djn\" (UID: \"67461890-6769-447f-af69-71f8d510e10c\") " pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:26.404285 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.404262 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6mz4\" (UniqueName: \"kubernetes.io/projected/67461890-6769-447f-af69-71f8d510e10c-kube-api-access-j6mz4\") pod \"payload-processing-external-model-7859b55bb7-k5djn\" (UID: \"67461890-6769-447f-af69-71f8d510e10c\") " pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:26.456069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.456043 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:26.761472 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:26.761441 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn"] Oct 30 09:48:26.761832 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:48:26.761806 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67461890_6769_447f_af69_71f8d510e10c.slice/crio-aa2c58953fdcfd2249e17549edf68a1fc21efded77416ee9e21b773d55ae9fa4 WatchSource:0}: Error finding container aa2c58953fdcfd2249e17549edf68a1fc21efded77416ee9e21b773d55ae9fa4: Status 404 returned error can't find the container with id aa2c58953fdcfd2249e17549edf68a1fc21efded77416ee9e21b773d55ae9fa4 Oct 30 09:48:27.233154 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.233113 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" event={"ID":"67461890-6769-447f-af69-71f8d510e10c","Type":"ContainerStarted","Data":"4ca843d65de752a0bc5c1d7497cf1a58f8e0c0855a4d1229f7e8bbaa8d4af34c"} Oct 30 09:48:27.233692 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.233176 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" event={"ID":"67461890-6769-447f-af69-71f8d510e10c","Type":"ContainerStarted","Data":"aa2c58953fdcfd2249e17549edf68a1fc21efded77416ee9e21b773d55ae9fa4"} Oct 30 09:48:27.233692 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.233321 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:27.277452 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.277402 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" podStartSLOduration=1.277387802 podStartE2EDuration="1.277387802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:48:27.274209959 +0000 UTC m=+3225.136915781" watchObservedRunningTime="2026-10-30 09:48:27.277387802 +0000 UTC m=+3225.140093624" Oct 30 09:48:27.314001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.313967 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx"] Oct 30 09:48:27.320130 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.320104 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:48:27.333775 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.333750 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx"] Oct 30 09:48:27.336273 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.336243 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"agc-cred-7655d9e07e16\"" Oct 30 09:48:27.336273 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.336245 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"routing-overlay\"" Oct 30 09:48:27.336448 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.336267 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"payload-processing-external-model-e2e--2110ea34-dockercfg-68rjw\"" Oct 30 09:48:27.336448 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.336307 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"kube-root-ca.crt\"" Oct 30 09:48:27.336448 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.336320 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d60c68\"" Oct 30 09:48:27.336448 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.336326 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"openshift-service-ca.crt\"" Oct 30 09:48:27.401417 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.401388 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgn6x\" (UniqueName: \"kubernetes.io/projected/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-kube-api-access-pgn6x\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx\" (UID: \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:48:27.401605 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.401531 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx\" (UID: \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:48:27.401681 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.401645 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx\" (UID: \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:48:27.401784 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.401769 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx\" (UID: \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:48:27.503676 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.503517 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx\" (UID: \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:48:27.503888 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.503679 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx\" (UID: \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:48:27.503888 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.503810 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx\" (UID: \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:48:27.504049 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.503953 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgn6x\" (UniqueName: \"kubernetes.io/projected/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-kube-api-access-pgn6x\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx\" (UID: \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:48:27.505674 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.505557 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx\" (UID: \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:48:27.506428 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.506403 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx\" (UID: \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:48:27.509147 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.509121 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx\" (UID: \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:48:27.529325 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.529295 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgn6x\" (UniqueName: \"kubernetes.io/projected/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-kube-api-access-pgn6x\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx\" (UID: \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:48:27.638265 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.638231 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:48:27.944516 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:27.944485 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx"] Oct 30 09:48:27.944702 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:48:27.944680 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ce89f9_6fe0_47a1_867e_40806e5aa6fb.slice/crio-513be8af0b82f6b0a3177a93771436c49544177782a26767a33f528f0da07551 WatchSource:0}: Error finding container 513be8af0b82f6b0a3177a93771436c49544177782a26767a33f528f0da07551: Status 404 returned error can't find the container with id 513be8af0b82f6b0a3177a93771436c49544177782a26767a33f528f0da07551 Oct 30 09:48:28.246352 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:28.246263 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" event={"ID":"98ce89f9-6fe0-47a1-867e-40806e5aa6fb","Type":"ContainerStarted","Data":"1cd7eb8c13d95fc3691cd9581bd733d6a51568de9235d870b97a987ce6932e20"} Oct 30 09:48:28.246352 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:28.246318 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" event={"ID":"98ce89f9-6fe0-47a1-867e-40806e5aa6fb","Type":"ContainerStarted","Data":"513be8af0b82f6b0a3177a93771436c49544177782a26767a33f528f0da07551"} Oct 30 09:48:28.246819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:28.246619 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:48:28.291604 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:28.291549 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" podStartSLOduration=1.291534707 podStartE2EDuration="1.291534707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:48:28.289012632 +0000 UTC m=+3226.151718455" watchObservedRunningTime="2026-10-30 09:48:28.291534707 +0000 UTC m=+3226.154240527" Oct 30 09:48:33.256157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:33.256125 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn"] Oct 30 09:48:33.259259 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:33.259237 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:35.613622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.613560 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6"] Oct 30 09:48:35.619774 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.619748 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:48:35.651607 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.638666 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6"] Oct 30 09:48:35.651607 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.647842 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d66d22\"/\"payload-processing-external-model-e2e--afb5c9ee-dockercfg-xbfdt\"" Oct 30 09:48:35.651607 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.647923 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d66d22\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d66d22\"" Oct 30 09:48:35.651607 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.647980 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d66d22\"/\"agc-cred-3ff3283a7fc1\"" Oct 30 09:48:35.651607 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.648030 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d66d22\"/\"openshift-service-ca.crt\"" Oct 30 09:48:35.651607 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.648103 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d66d22\"/\"kube-root-ca.crt\"" Oct 30 09:48:35.651607 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.648153 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d66d22\"/\"routing-overlay\"" Oct 30 09:48:35.761899 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.761870 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67kb8\" (UniqueName: \"kubernetes.io/projected/f935fb8b-e17c-452c-a830-279e0f52ad1e-kube-api-access-67kb8\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6\" (UID: \"f935fb8b-e17c-452c-a830-279e0f52ad1e\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:48:35.762109 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.762078 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f935fb8b-e17c-452c-a830-279e0f52ad1e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6\" (UID: \"f935fb8b-e17c-452c-a830-279e0f52ad1e\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:48:35.762258 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.762182 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f935fb8b-e17c-452c-a830-279e0f52ad1e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6\" (UID: \"f935fb8b-e17c-452c-a830-279e0f52ad1e\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:48:35.762258 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.762231 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f935fb8b-e17c-452c-a830-279e0f52ad1e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6\" (UID: \"f935fb8b-e17c-452c-a830-279e0f52ad1e\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:48:35.864113 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.864003 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f935fb8b-e17c-452c-a830-279e0f52ad1e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6\" (UID: \"f935fb8b-e17c-452c-a830-279e0f52ad1e\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:48:35.864310 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.864116 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f935fb8b-e17c-452c-a830-279e0f52ad1e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6\" (UID: \"f935fb8b-e17c-452c-a830-279e0f52ad1e\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:48:35.864310 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.864178 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67kb8\" (UniqueName: \"kubernetes.io/projected/f935fb8b-e17c-452c-a830-279e0f52ad1e-kube-api-access-67kb8\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6\" (UID: \"f935fb8b-e17c-452c-a830-279e0f52ad1e\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:48:35.864440 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.864407 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f935fb8b-e17c-452c-a830-279e0f52ad1e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6\" (UID: \"f935fb8b-e17c-452c-a830-279e0f52ad1e\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:48:35.865233 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.865193 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f935fb8b-e17c-452c-a830-279e0f52ad1e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6\" (UID: \"f935fb8b-e17c-452c-a830-279e0f52ad1e\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:48:35.865233 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.865221 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f935fb8b-e17c-452c-a830-279e0f52ad1e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6\" (UID: \"f935fb8b-e17c-452c-a830-279e0f52ad1e\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:48:35.867362 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.867340 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f935fb8b-e17c-452c-a830-279e0f52ad1e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6\" (UID: \"f935fb8b-e17c-452c-a830-279e0f52ad1e\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:48:35.886218 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.886191 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67kb8\" (UniqueName: \"kubernetes.io/projected/f935fb8b-e17c-452c-a830-279e0f52ad1e-kube-api-access-67kb8\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6\" (UID: \"f935fb8b-e17c-452c-a830-279e0f52ad1e\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:48:35.935279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:35.935253 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:48:36.249260 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:36.249233 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6"] Oct 30 09:48:36.249723 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:48:36.249690 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf935fb8b_e17c_452c_a830_279e0f52ad1e.slice/crio-6353e3a186d8549b57d4ce2c3314def469ff5733a49847441911f4b5acc5fc54 WatchSource:0}: Error finding container 6353e3a186d8549b57d4ce2c3314def469ff5733a49847441911f4b5acc5fc54: Status 404 returned error can't find the container with id 6353e3a186d8549b57d4ce2c3314def469ff5733a49847441911f4b5acc5fc54 Oct 30 09:48:36.333921 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:36.333892 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" event={"ID":"f935fb8b-e17c-452c-a830-279e0f52ad1e","Type":"ContainerStarted","Data":"6353e3a186d8549b57d4ce2c3314def469ff5733a49847441911f4b5acc5fc54"} Oct 30 09:48:37.344331 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:37.344289 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" event={"ID":"f935fb8b-e17c-452c-a830-279e0f52ad1e","Type":"ContainerStarted","Data":"c454d642818b1da422cd7aa82bf1e3a8117f624186b32d419fb231ef537b2110"} Oct 30 09:48:37.344756 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:37.344383 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:48:37.386834 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:37.386779 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" podStartSLOduration=2.386766171 podStartE2EDuration="2.386766171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:48:37.385167019 +0000 UTC m=+3235.247872843" watchObservedRunningTime="2026-10-30 09:48:37.386766171 +0000 UTC m=+3235.249471992" Oct 30 09:48:38.257427 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:38.257386 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" podUID="67461890-6769-447f-af69-71f8d510e10c" containerName="payload-processing" containerID="cri-o://4ca843d65de752a0bc5c1d7497cf1a58f8e0c0855a4d1229f7e8bbaa8d4af34c" gracePeriod=25 Oct 30 09:48:38.533094 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:38.533070 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:38.621806 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:38.621767 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67461890-6769-447f-af69-71f8d510e10c-kube-api-access-j6mz4\" (UniqueName: \"kubernetes.io/projected/67461890-6769-447f-af69-71f8d510e10c-kube-api-access-j6mz4\") pod \"67461890-6769-447f-af69-71f8d510e10c\" (UID: \"67461890-6769-447f-af69-71f8d510e10c\") " Oct 30 09:48:38.621987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:38.621832 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67461890-6769-447f-af69-71f8d510e10c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/67461890-6769-447f-af69-71f8d510e10c-provider-credentials\") pod \"67461890-6769-447f-af69-71f8d510e10c\" (UID: \"67461890-6769-447f-af69-71f8d510e10c\") " Oct 30 09:48:38.621987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:38.621904 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67461890-6769-447f-af69-71f8d510e10c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67461890-6769-447f-af69-71f8d510e10c-routing-overlay-volume\") pod \"67461890-6769-447f-af69-71f8d510e10c\" (UID: \"67461890-6769-447f-af69-71f8d510e10c\") " Oct 30 09:48:38.622130 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:38.621991 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67461890-6769-447f-af69-71f8d510e10c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67461890-6769-447f-af69-71f8d510e10c-plugins-config-volume\") pod \"67461890-6769-447f-af69-71f8d510e10c\" (UID: \"67461890-6769-447f-af69-71f8d510e10c\") " Oct 30 09:48:38.622373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:38.622339 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67461890-6769-447f-af69-71f8d510e10c-routing-overlay-volume" pod "67461890-6769-447f-af69-71f8d510e10c" (UID: "67461890-6769-447f-af69-71f8d510e10c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:48:38.622518 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:38.622405 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67461890-6769-447f-af69-71f8d510e10c-plugins-config-volume" pod "67461890-6769-447f-af69-71f8d510e10c" (UID: "67461890-6769-447f-af69-71f8d510e10c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:48:38.622695 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:38.622674 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67461890-6769-447f-af69-71f8d510e10c-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:38.622767 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:38.622712 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67461890-6769-447f-af69-71f8d510e10c-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:38.624276 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:38.624254 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67461890-6769-447f-af69-71f8d510e10c-kube-api-access-j6mz4" pod "67461890-6769-447f-af69-71f8d510e10c" (UID: "67461890-6769-447f-af69-71f8d510e10c"). InnerVolumeSpecName "kube-api-access-j6mz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:48:38.624650 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:38.624631 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67461890-6769-447f-af69-71f8d510e10c-provider-credentials" pod "67461890-6769-447f-af69-71f8d510e10c" (UID: "67461890-6769-447f-af69-71f8d510e10c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:48:38.724567 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:38.724538 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6mz4\" (UniqueName: \"kubernetes.io/projected/67461890-6769-447f-af69-71f8d510e10c-kube-api-access-j6mz4\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:38.724567 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:38.724564 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67461890-6769-447f-af69-71f8d510e10c-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:39.256130 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:39.256104 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:48:39.362789 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:39.362752 5938 generic.go:356] "Generic (PLEG): container finished" podID="67461890-6769-447f-af69-71f8d510e10c" containerID="4ca843d65de752a0bc5c1d7497cf1a58f8e0c0855a4d1229f7e8bbaa8d4af34c" exitCode=0 Oct 30 09:48:39.362976 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:39.362813 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" Oct 30 09:48:39.362976 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:39.362832 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" event={"ID":"67461890-6769-447f-af69-71f8d510e10c","Type":"ContainerDied","Data":"4ca843d65de752a0bc5c1d7497cf1a58f8e0c0855a4d1229f7e8bbaa8d4af34c"} Oct 30 09:48:39.362976 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:39.362891 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn" event={"ID":"67461890-6769-447f-af69-71f8d510e10c","Type":"ContainerDied","Data":"aa2c58953fdcfd2249e17549edf68a1fc21efded77416ee9e21b773d55ae9fa4"} Oct 30 09:48:39.362976 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:39.362916 5938 scope.go:122] "RemoveContainer" containerID="4ca843d65de752a0bc5c1d7497cf1a58f8e0c0855a4d1229f7e8bbaa8d4af34c" Oct 30 09:48:39.374665 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:39.374647 5938 scope.go:122] "RemoveContainer" containerID="4ca843d65de752a0bc5c1d7497cf1a58f8e0c0855a4d1229f7e8bbaa8d4af34c" Oct 30 09:48:39.374929 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:39.374906 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca843d65de752a0bc5c1d7497cf1a58f8e0c0855a4d1229f7e8bbaa8d4af34c\": container with ID starting with 4ca843d65de752a0bc5c1d7497cf1a58f8e0c0855a4d1229f7e8bbaa8d4af34c not found: ID does not exist" containerID="4ca843d65de752a0bc5c1d7497cf1a58f8e0c0855a4d1229f7e8bbaa8d4af34c" Oct 30 09:48:39.375026 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:39.374936 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca843d65de752a0bc5c1d7497cf1a58f8e0c0855a4d1229f7e8bbaa8d4af34c"} err="failed to get container status \"4ca843d65de752a0bc5c1d7497cf1a58f8e0c0855a4d1229f7e8bbaa8d4af34c\": rpc error: code = NotFound desc = could not find container \"4ca843d65de752a0bc5c1d7497cf1a58f8e0c0855a4d1229f7e8bbaa8d4af34c\": container with ID starting with 4ca843d65de752a0bc5c1d7497cf1a58f8e0c0855a4d1229f7e8bbaa8d4af34c not found: ID does not exist" Oct 30 09:48:39.421904 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:39.421877 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn"] Oct 30 09:48:39.430561 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:39.430538 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7859b55bb7-k5djn"] Oct 30 09:48:40.258875 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:40.258847 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67461890-6769-447f-af69-71f8d510e10c" path="/var/lib/kubelet/pods/67461890-6769-447f-af69-71f8d510e10c/volumes" Oct 30 09:48:44.051357 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:44.051328 5938 scope.go:122] "RemoveContainer" containerID="83b4c6b1b762e42611fe8c204a24934203f5d7fe439a63bd1c3356f8404238bd" Oct 30 09:48:45.815456 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:48:45.815375 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:48:48.354877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:48.354847 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:48:49.381855 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.381819 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht"] Oct 30 09:48:49.382857 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.382838 5938 cpu_manager.go:414] "containerMap: removing container" podUID="67461890-6769-447f-af69-71f8d510e10c" containerName="payload-processing" Oct 30 09:48:49.382857 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.382858 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67461890-6769-447f-af69-71f8d510e10c" containerName="payload-processing" Oct 30 09:48:49.383061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.382984 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="67461890-6769-447f-af69-71f8d510e10c" containerName="payload-processing" Oct 30 09:48:49.388465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.388446 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" Oct 30 09:48:49.404895 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.404866 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht"] Oct 30 09:48:49.416831 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.413242 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:48:49.416831 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.413349 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:48:49.508532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.508498 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf5c897-be06-4cf9-9bdd-33974bd190e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6481d0bf-kmjht\" (UID: \"bdf5c897-be06-4cf9-9bdd-33974bd190e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" Oct 30 09:48:49.508712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.508661 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqfwx\" (UniqueName: \"kubernetes.io/projected/bdf5c897-be06-4cf9-9bdd-33974bd190e0-kube-api-access-lqfwx\") pod \"maas-api-revoke-sub-6481d0bf-kmjht\" (UID: \"bdf5c897-be06-4cf9-9bdd-33974bd190e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" Oct 30 09:48:49.508712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.508701 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf5c897-be06-4cf9-9bdd-33974bd190e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6481d0bf-kmjht\" (UID: \"bdf5c897-be06-4cf9-9bdd-33974bd190e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" Oct 30 09:48:49.609917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.609885 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf5c897-be06-4cf9-9bdd-33974bd190e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6481d0bf-kmjht\" (UID: \"bdf5c897-be06-4cf9-9bdd-33974bd190e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" Oct 30 09:48:49.610087 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.610033 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqfwx\" (UniqueName: \"kubernetes.io/projected/bdf5c897-be06-4cf9-9bdd-33974bd190e0-kube-api-access-lqfwx\") pod \"maas-api-revoke-sub-6481d0bf-kmjht\" (UID: \"bdf5c897-be06-4cf9-9bdd-33974bd190e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" Oct 30 09:48:49.610176 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.610090 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf5c897-be06-4cf9-9bdd-33974bd190e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6481d0bf-kmjht\" (UID: \"bdf5c897-be06-4cf9-9bdd-33974bd190e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" Oct 30 09:48:49.610787 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.610738 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf5c897-be06-4cf9-9bdd-33974bd190e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6481d0bf-kmjht\" (UID: \"bdf5c897-be06-4cf9-9bdd-33974bd190e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" Oct 30 09:48:49.612457 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.612428 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf5c897-be06-4cf9-9bdd-33974bd190e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6481d0bf-kmjht\" (UID: \"bdf5c897-be06-4cf9-9bdd-33974bd190e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" Oct 30 09:48:49.642960 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.642907 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqfwx\" (UniqueName: \"kubernetes.io/projected/bdf5c897-be06-4cf9-9bdd-33974bd190e0-kube-api-access-lqfwx\") pod \"maas-api-revoke-sub-6481d0bf-kmjht\" (UID: \"bdf5c897-be06-4cf9-9bdd-33974bd190e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" Oct 30 09:48:49.705745 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:49.705727 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" Oct 30 09:48:50.008840 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:50.008805 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht"] Oct 30 09:48:50.009037 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:48:50.009014 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf5c897_be06_4cf9_9bdd_33974bd190e0.slice/crio-90d80dd8b1e2b7e2ebfa2d28ed23e933275f92b4abb9affac9aee681376de9ea WatchSource:0}: Error finding container 90d80dd8b1e2b7e2ebfa2d28ed23e933275f92b4abb9affac9aee681376de9ea: Status 404 returned error can't find the container with id 90d80dd8b1e2b7e2ebfa2d28ed23e933275f92b4abb9affac9aee681376de9ea Oct 30 09:48:50.475391 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:50.475362 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" event={"ID":"bdf5c897-be06-4cf9-9bdd-33974bd190e0","Type":"ContainerStarted","Data":"b04f9bf46449ae38e351ca350a1d7fa1c39a7291c616ad7b86a32780d2a4c104"} Oct 30 09:48:50.475879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:50.475413 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" event={"ID":"bdf5c897-be06-4cf9-9bdd-33974bd190e0","Type":"ContainerStarted","Data":"90d80dd8b1e2b7e2ebfa2d28ed23e933275f92b4abb9affac9aee681376de9ea"} Oct 30 09:48:50.520789 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:50.520743 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" podStartSLOduration=1.520729845 podStartE2EDuration="1.520729845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:48:50.517525288 +0000 UTC m=+3248.380231112" watchObservedRunningTime="2026-10-30 09:48:50.520729845 +0000 UTC m=+3248.383435666" Oct 30 09:48:51.484259 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.484228 5938 generic.go:356] "Generic (PLEG): container finished" podID="bdf5c897-be06-4cf9-9bdd-33974bd190e0" containerID="b04f9bf46449ae38e351ca350a1d7fa1c39a7291c616ad7b86a32780d2a4c104" exitCode=0 Oct 30 09:48:51.484690 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.484281 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" event={"ID":"bdf5c897-be06-4cf9-9bdd-33974bd190e0","Type":"ContainerDied","Data":"b04f9bf46449ae38e351ca350a1d7fa1c39a7291c616ad7b86a32780d2a4c104"} Oct 30 09:48:51.581342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.581308 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd"] Oct 30 09:48:51.585860 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.585837 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.604980 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.604886 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 09:48:51.604980 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.604907 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 09:48:51.608735 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.608715 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 09:48:51.609799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.609780 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h44vk\"" Oct 30 09:48:51.610255 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.610232 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd"] Oct 30 09:48:51.638389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.638365 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.638530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.638517 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcn2r\" (UniqueName: \"kubernetes.io/projected/5a094c60-d6c5-4f30-8085-6d8381eb70b2-kube-api-access-gcn2r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.638684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.638668 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.638786 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.638709 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.638786 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.638738 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.638940 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.638829 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.638940 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.638901 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a094c60-d6c5-4f30-8085-6d8381eb70b2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.740134 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.740073 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcn2r\" (UniqueName: \"kubernetes.io/projected/5a094c60-d6c5-4f30-8085-6d8381eb70b2-kube-api-access-gcn2r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.740242 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.740176 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.740242 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.740207 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.740242 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.740231 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.740416 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.740276 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.740416 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.740308 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a094c60-d6c5-4f30-8085-6d8381eb70b2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.740416 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.740366 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.741047 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.741018 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.742983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.742962 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.743089 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.743010 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a094c60-d6c5-4f30-8085-6d8381eb70b2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.743089 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.743021 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.743217 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.743180 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.761495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.761462 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcn2r\" (UniqueName: \"kubernetes.io/projected/5a094c60-d6c5-4f30-8085-6d8381eb70b2-kube-api-access-gcn2r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.761568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.761547 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z6xd\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:51.900865 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:51.900840 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:52.220553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.220527 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd"] Oct 30 09:48:52.221900 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:48:52.221875 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a094c60_d6c5_4f30_8085_6d8381eb70b2.slice/crio-c5eb8396c75c5a55d808cd65d5d10e3b869928f502d171b67ba13b7cee66017b WatchSource:0}: Error finding container c5eb8396c75c5a55d808cd65d5d10e3b869928f502d171b67ba13b7cee66017b: Status 404 returned error can't find the container with id c5eb8396c75c5a55d808cd65d5d10e3b869928f502d171b67ba13b7cee66017b Oct 30 09:48:52.226026 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.225984 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:48:52.226123 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.226069 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:48:52.494693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.494573 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" event={"ID":"5a094c60-d6c5-4f30-8085-6d8381eb70b2","Type":"ContainerStarted","Data":"1f024f1965b3cf41a3866bc9919b5e45d0734c4267a20c01fa0ff5a9e8b595ac"} Oct 30 09:48:52.494693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.494654 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:48:52.494693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.494677 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" event={"ID":"5a094c60-d6c5-4f30-8085-6d8381eb70b2","Type":"ContainerStarted","Data":"c5eb8396c75c5a55d808cd65d5d10e3b869928f502d171b67ba13b7cee66017b"} Oct 30 09:48:52.581990 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.581920 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" podStartSLOduration=1.5818948210000001 podStartE2EDuration="1.581894821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:48:52.551547552 +0000 UTC m=+3250.414253378" watchObservedRunningTime="2026-10-30 09:48:52.581894821 +0000 UTC m=+3250.444600644" Oct 30 09:48:52.648071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.648037 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" Oct 30 09:48:52.755790 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.754156 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdf5c897-be06-4cf9-9bdd-33974bd190e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf5c897-be06-4cf9-9bdd-33974bd190e0-maas-api-xks-ca\") pod \"bdf5c897-be06-4cf9-9bdd-33974bd190e0\" (UID: \"bdf5c897-be06-4cf9-9bdd-33974bd190e0\") " Oct 30 09:48:52.758373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.757787 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf5c897-be06-4cf9-9bdd-33974bd190e0-kube-api-access-lqfwx\" (UniqueName: \"kubernetes.io/projected/bdf5c897-be06-4cf9-9bdd-33974bd190e0-kube-api-access-lqfwx\") pod \"bdf5c897-be06-4cf9-9bdd-33974bd190e0\" (UID: \"bdf5c897-be06-4cf9-9bdd-33974bd190e0\") " Oct 30 09:48:52.758373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.757938 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdf5c897-be06-4cf9-9bdd-33974bd190e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf5c897-be06-4cf9-9bdd-33974bd190e0-maas-api-openshift-ca\") pod \"bdf5c897-be06-4cf9-9bdd-33974bd190e0\" (UID: \"bdf5c897-be06-4cf9-9bdd-33974bd190e0\") " Oct 30 09:48:52.758943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.758894 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf5c897-be06-4cf9-9bdd-33974bd190e0-maas-api-openshift-ca" pod "bdf5c897-be06-4cf9-9bdd-33974bd190e0" (UID: "bdf5c897-be06-4cf9-9bdd-33974bd190e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:48:52.760526 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.760498 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf5c897-be06-4cf9-9bdd-33974bd190e0-maas-api-xks-ca" pod "bdf5c897-be06-4cf9-9bdd-33974bd190e0" (UID: "bdf5c897-be06-4cf9-9bdd-33974bd190e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:48:52.764249 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.764224 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf5c897-be06-4cf9-9bdd-33974bd190e0-kube-api-access-lqfwx" pod "bdf5c897-be06-4cf9-9bdd-33974bd190e0" (UID: "bdf5c897-be06-4cf9-9bdd-33974bd190e0"). InnerVolumeSpecName "kube-api-access-lqfwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:48:52.860082 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.860045 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf5c897-be06-4cf9-9bdd-33974bd190e0-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:52.860082 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.860079 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqfwx\" (UniqueName: \"kubernetes.io/projected/bdf5c897-be06-4cf9-9bdd-33974bd190e0-kube-api-access-lqfwx\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:52.860082 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:52.860091 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf5c897-be06-4cf9-9bdd-33974bd190e0-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:48:53.505345 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:53.505315 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" Oct 30 09:48:53.505886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:53.505349 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht" event={"ID":"bdf5c897-be06-4cf9-9bdd-33974bd190e0","Type":"ContainerDied","Data":"90d80dd8b1e2b7e2ebfa2d28ed23e933275f92b4abb9affac9aee681376de9ea"} Oct 30 09:48:53.505886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:53.505396 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90d80dd8b1e2b7e2ebfa2d28ed23e933275f92b4abb9affac9aee681376de9ea" Oct 30 09:48:53.508500 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:53.508474 5938 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9z6xd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:48:53.508693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:53.508527 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" podUID="5a094c60-d6c5-4f30-8085-6d8381eb70b2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:48:56.506542 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:48:56.506513 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:49:04.508202 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:04.508161 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv"] Oct 30 09:49:04.508913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:04.508428 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" podUID="80566eca-1f41-450c-968e-f917a76cc203" containerName="istio-proxy" containerID="cri-o://30c18541d9eef9767ebae746842723c6b55110bbb8ddb7f257e4b44a37a1b7d8" gracePeriod=30 Oct 30 09:49:09.345879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.345843 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:49:09.346288 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.345913 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" podUID="80566eca-1f41-450c-968e-f917a76cc203" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:49:09.692441 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.692339 5938 generic.go:356] "Generic (PLEG): container finished" podID="80566eca-1f41-450c-968e-f917a76cc203" containerID="30c18541d9eef9767ebae746842723c6b55110bbb8ddb7f257e4b44a37a1b7d8" exitCode=0 Oct 30 09:49:09.692627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.692524 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" event={"ID":"80566eca-1f41-450c-968e-f917a76cc203","Type":"ContainerDied","Data":"30c18541d9eef9767ebae746842723c6b55110bbb8ddb7f257e4b44a37a1b7d8"} Oct 30 09:49:09.784665 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.784642 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:49:09.880430 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.880354 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80566eca-1f41-450c-968e-f917a76cc203-kube-api-access-v4bnw\" (UniqueName: \"kubernetes.io/projected/80566eca-1f41-450c-968e-f917a76cc203-kube-api-access-v4bnw\") pod \"80566eca-1f41-450c-968e-f917a76cc203\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " Oct 30 09:49:09.880430 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.880410 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80566eca-1f41-450c-968e-f917a76cc203-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80566eca-1f41-450c-968e-f917a76cc203-istio-podinfo\") pod \"80566eca-1f41-450c-968e-f917a76cc203\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " Oct 30 09:49:09.880732 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.880494 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80566eca-1f41-450c-968e-f917a76cc203-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80566eca-1f41-450c-968e-f917a76cc203-istiod-ca-cert\") pod \"80566eca-1f41-450c-968e-f917a76cc203\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " Oct 30 09:49:09.880732 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.880658 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-istio-envoy\") pod \"80566eca-1f41-450c-968e-f917a76cc203\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " Oct 30 09:49:09.880862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.880737 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-credential-socket\") pod \"80566eca-1f41-450c-968e-f917a76cc203\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " Oct 30 09:49:09.880862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.880823 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-workload-certs\") pod \"80566eca-1f41-450c-968e-f917a76cc203\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " Oct 30 09:49:09.880862 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.880848 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80566eca-1f41-450c-968e-f917a76cc203-istio-token\" (UniqueName: \"kubernetes.io/projected/80566eca-1f41-450c-968e-f917a76cc203-istio-token\") pod \"80566eca-1f41-450c-968e-f917a76cc203\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " Oct 30 09:49:09.881117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.880919 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-workload-socket\") pod \"80566eca-1f41-450c-968e-f917a76cc203\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " Oct 30 09:49:09.881117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.880977 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-istio-data\") pod \"80566eca-1f41-450c-968e-f917a76cc203\" (UID: \"80566eca-1f41-450c-968e-f917a76cc203\") " Oct 30 09:49:09.881237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.881205 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-workload-certs" pod "80566eca-1f41-450c-968e-f917a76cc203" (UID: "80566eca-1f41-450c-968e-f917a76cc203"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:49:09.881412 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.881390 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-istio-data" pod "80566eca-1f41-450c-968e-f917a76cc203" (UID: "80566eca-1f41-450c-968e-f917a76cc203"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:49:09.881529 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.881462 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-credential-socket" pod "80566eca-1f41-450c-968e-f917a76cc203" (UID: "80566eca-1f41-450c-968e-f917a76cc203"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:49:09.881635 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.881612 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-workload-socket" pod "80566eca-1f41-450c-968e-f917a76cc203" (UID: "80566eca-1f41-450c-968e-f917a76cc203"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:49:09.881870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.881652 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80566eca-1f41-450c-968e-f917a76cc203-istiod-ca-cert" pod "80566eca-1f41-450c-968e-f917a76cc203" (UID: "80566eca-1f41-450c-968e-f917a76cc203"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:49:09.881870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.881739 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:49:09.881870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.881760 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:49:09.881870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.881774 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:49:09.884507 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.884480 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80566eca-1f41-450c-968e-f917a76cc203-istio-podinfo" pod "80566eca-1f41-450c-968e-f917a76cc203" (UID: "80566eca-1f41-450c-968e-f917a76cc203"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:49:09.884630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.884534 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-istio-envoy" pod "80566eca-1f41-450c-968e-f917a76cc203" (UID: "80566eca-1f41-450c-968e-f917a76cc203"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:49:09.884630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.884575 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80566eca-1f41-450c-968e-f917a76cc203-kube-api-access-v4bnw" pod "80566eca-1f41-450c-968e-f917a76cc203" (UID: "80566eca-1f41-450c-968e-f917a76cc203"). InnerVolumeSpecName "kube-api-access-v4bnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:49:09.885985 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.885960 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80566eca-1f41-450c-968e-f917a76cc203-istio-token" pod "80566eca-1f41-450c-968e-f917a76cc203" (UID: "80566eca-1f41-450c-968e-f917a76cc203"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:49:09.983142 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.983110 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4bnw\" (UniqueName: \"kubernetes.io/projected/80566eca-1f41-450c-968e-f917a76cc203-kube-api-access-v4bnw\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:49:09.983142 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.983137 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80566eca-1f41-450c-968e-f917a76cc203-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:49:09.983142 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.983147 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80566eca-1f41-450c-968e-f917a76cc203-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:49:09.983367 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.983156 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:49:09.983367 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.983163 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80566eca-1f41-450c-968e-f917a76cc203-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:49:09.983367 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:09.983171 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80566eca-1f41-450c-968e-f917a76cc203-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:49:10.701435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:10.701403 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" Oct 30 09:49:10.701892 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:10.701433 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv" event={"ID":"80566eca-1f41-450c-968e-f917a76cc203","Type":"ContainerDied","Data":"5878460f70d6dfb49935c9ef56c48a1e85debf9df235598511b14ac998eb21cb"} Oct 30 09:49:10.701892 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:10.701511 5938 scope.go:122] "RemoveContainer" containerID="30c18541d9eef9767ebae746842723c6b55110bbb8ddb7f257e4b44a37a1b7d8" Oct 30 09:49:10.766069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:10.766025 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv"] Oct 30 09:49:10.779332 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:10.779303 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-wbtlv"] Oct 30 09:49:12.259243 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:12.259202 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80566eca-1f41-450c-968e-f917a76cc203" path="/var/lib/kubelet/pods/80566eca-1f41-450c-968e-f917a76cc203/volumes" Oct 30 09:49:14.029546 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:14.029513 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v"] Oct 30 09:49:15.808759 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:49:15.808726 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:49:19.030807 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.030771 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" podUID="eb642f5e-4496-480d-a324-06c0df2fe3bb" containerName="payload-processing" containerID="cri-o://5fcd684aa0e4314ce89f2b1c906e5d38fcd9a8af5a95f5343563a4afb6266a70" gracePeriod=25 Oct 30 09:49:19.313093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.313066 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:49:19.414104 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.414067 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb642f5e-4496-480d-a324-06c0df2fe3bb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb642f5e-4496-480d-a324-06c0df2fe3bb-provider-credentials\") pod \"eb642f5e-4496-480d-a324-06c0df2fe3bb\" (UID: \"eb642f5e-4496-480d-a324-06c0df2fe3bb\") " Oct 30 09:49:19.414282 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.414152 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb642f5e-4496-480d-a324-06c0df2fe3bb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb642f5e-4496-480d-a324-06c0df2fe3bb-routing-overlay-volume\") pod \"eb642f5e-4496-480d-a324-06c0df2fe3bb\" (UID: \"eb642f5e-4496-480d-a324-06c0df2fe3bb\") " Oct 30 09:49:19.414282 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.414199 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb642f5e-4496-480d-a324-06c0df2fe3bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb642f5e-4496-480d-a324-06c0df2fe3bb-plugins-config-volume\") pod \"eb642f5e-4496-480d-a324-06c0df2fe3bb\" (UID: \"eb642f5e-4496-480d-a324-06c0df2fe3bb\") " Oct 30 09:49:19.414282 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.414259 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb642f5e-4496-480d-a324-06c0df2fe3bb-kube-api-access-j8tql\" (UniqueName: \"kubernetes.io/projected/eb642f5e-4496-480d-a324-06c0df2fe3bb-kube-api-access-j8tql\") pod \"eb642f5e-4496-480d-a324-06c0df2fe3bb\" (UID: \"eb642f5e-4496-480d-a324-06c0df2fe3bb\") " Oct 30 09:49:19.414599 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.414560 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb642f5e-4496-480d-a324-06c0df2fe3bb-routing-overlay-volume" pod "eb642f5e-4496-480d-a324-06c0df2fe3bb" (UID: "eb642f5e-4496-480d-a324-06c0df2fe3bb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:49:19.414684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.414626 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb642f5e-4496-480d-a324-06c0df2fe3bb-plugins-config-volume" pod "eb642f5e-4496-480d-a324-06c0df2fe3bb" (UID: "eb642f5e-4496-480d-a324-06c0df2fe3bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:49:19.414847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.414832 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb642f5e-4496-480d-a324-06c0df2fe3bb-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:49:19.414894 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.414852 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb642f5e-4496-480d-a324-06c0df2fe3bb-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:49:19.416370 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.416347 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb642f5e-4496-480d-a324-06c0df2fe3bb-provider-credentials" pod "eb642f5e-4496-480d-a324-06c0df2fe3bb" (UID: "eb642f5e-4496-480d-a324-06c0df2fe3bb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:49:19.416524 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.416505 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb642f5e-4496-480d-a324-06c0df2fe3bb-kube-api-access-j8tql" pod "eb642f5e-4496-480d-a324-06c0df2fe3bb" (UID: "eb642f5e-4496-480d-a324-06c0df2fe3bb"). InnerVolumeSpecName "kube-api-access-j8tql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:49:19.515837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.515808 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb642f5e-4496-480d-a324-06c0df2fe3bb-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:49:19.515837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.515836 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8tql\" (UniqueName: \"kubernetes.io/projected/eb642f5e-4496-480d-a324-06c0df2fe3bb-kube-api-access-j8tql\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:49:19.770706 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.770677 5938 generic.go:356] "Generic (PLEG): container finished" podID="eb642f5e-4496-480d-a324-06c0df2fe3bb" containerID="5fcd684aa0e4314ce89f2b1c906e5d38fcd9a8af5a95f5343563a4afb6266a70" exitCode=0 Oct 30 09:49:19.770872 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.770719 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" Oct 30 09:49:19.770872 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.770753 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" event={"ID":"eb642f5e-4496-480d-a324-06c0df2fe3bb","Type":"ContainerDied","Data":"5fcd684aa0e4314ce89f2b1c906e5d38fcd9a8af5a95f5343563a4afb6266a70"} Oct 30 09:49:19.770872 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.770796 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v" event={"ID":"eb642f5e-4496-480d-a324-06c0df2fe3bb","Type":"ContainerDied","Data":"0e8841cb2b75e43345fbbd29b895220044d589b8f166d196d6479f30cb71e274"} Oct 30 09:49:19.770872 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.770821 5938 scope.go:122] "RemoveContainer" containerID="5fcd684aa0e4314ce89f2b1c906e5d38fcd9a8af5a95f5343563a4afb6266a70" Oct 30 09:49:19.782426 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.782409 5938 scope.go:122] "RemoveContainer" containerID="5fcd684aa0e4314ce89f2b1c906e5d38fcd9a8af5a95f5343563a4afb6266a70" Oct 30 09:49:19.782808 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:49:19.782788 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fcd684aa0e4314ce89f2b1c906e5d38fcd9a8af5a95f5343563a4afb6266a70\": container with ID starting with 5fcd684aa0e4314ce89f2b1c906e5d38fcd9a8af5a95f5343563a4afb6266a70 not found: ID does not exist" containerID="5fcd684aa0e4314ce89f2b1c906e5d38fcd9a8af5a95f5343563a4afb6266a70" Oct 30 09:49:19.782864 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.782815 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fcd684aa0e4314ce89f2b1c906e5d38fcd9a8af5a95f5343563a4afb6266a70"} err="failed to get container status \"5fcd684aa0e4314ce89f2b1c906e5d38fcd9a8af5a95f5343563a4afb6266a70\": rpc error: code = NotFound desc = could not find container \"5fcd684aa0e4314ce89f2b1c906e5d38fcd9a8af5a95f5343563a4afb6266a70\": container with ID starting with 5fcd684aa0e4314ce89f2b1c906e5d38fcd9a8af5a95f5343563a4afb6266a70 not found: ID does not exist" Oct 30 09:49:19.828225 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.828204 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v"] Oct 30 09:49:19.841815 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:19.841798 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-58fbcd848f-ssc8v"] Oct 30 09:49:20.267612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:20.266958 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb642f5e-4496-480d-a324-06c0df2fe3bb" path="/var/lib/kubelet/pods/eb642f5e-4496-480d-a324-06c0df2fe3bb/volumes" Oct 30 09:49:28.031780 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:28.031739 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs"] Oct 30 09:49:28.044773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:28.044744 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de59b49-8jkxs"] Oct 30 09:49:28.268902 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:28.268824 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e82e336-f8ad-43f2-a417-32432fa7192c" path="/var/lib/kubelet/pods/6e82e336-f8ad-43f2-a417-32432fa7192c/volumes" Oct 30 09:49:40.028853 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:40.028822 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd"] Oct 30 09:49:40.041970 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:40.041940 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81554c57-sdzdd"] Oct 30 09:49:40.258181 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:40.258147 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5415e392-4927-49a4-a271-ddd0487cb071" path="/var/lib/kubelet/pods/5415e392-4927-49a4-a271-ddd0487cb071/volumes" Oct 30 09:49:44.115512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:44.115477 5938 scope.go:122] "RemoveContainer" containerID="0d243b6e97b4b8418c50e0cd57187a56b72a3721fa5125931392e5665d2fb2cc" Oct 30 09:49:44.125778 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:44.125757 5938 scope.go:122] "RemoveContainer" containerID="44b0ba5e14b69a4e8df559a23f03c1b54f2ab61fc76f4e07a11788856d00e44f" Oct 30 09:49:45.808763 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:49:45.808722 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:49:52.035014 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:52.034979 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl"] Oct 30 09:49:52.047666 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:52.047640 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-675f9f2b-xdlfl"] Oct 30 09:49:52.259568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:49:52.259536 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f08e92f2-4e98-4e05-936f-e11730ed863e" path="/var/lib/kubelet/pods/f08e92f2-4e98-4e05-936f-e11730ed863e/volumes" Oct 30 09:50:15.808704 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:50:15.808669 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:50:44.167363 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:44.167326 5938 scope.go:122] "RemoveContainer" containerID="fb76de80a843f5a268a328b67e5d508c78bf213947b8b39a2936208db892955e" Oct 30 09:50:44.178081 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:44.178061 5938 scope.go:122] "RemoveContainer" containerID="1eb89d0d532a063e5b06caf7c160aeb58e34e493d34b7c5dd5c80f4fadc32a04" Oct 30 09:50:45.810386 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:50:45.810350 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:50:49.245100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.245067 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz"] Oct 30 09:50:49.245804 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.245786 5938 cpu_manager.go:414] "containerMap: removing container" podUID="eb642f5e-4496-480d-a324-06c0df2fe3bb" containerName="payload-processing" Oct 30 09:50:49.245804 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.245803 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb642f5e-4496-480d-a324-06c0df2fe3bb" containerName="payload-processing" Oct 30 09:50:49.245975 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.245830 5938 cpu_manager.go:414] "containerMap: removing container" podUID="bdf5c897-be06-4cf9-9bdd-33974bd190e0" containerName="revoke-keys" Oct 30 09:50:49.245975 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.245836 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf5c897-be06-4cf9-9bdd-33974bd190e0" containerName="revoke-keys" Oct 30 09:50:49.245975 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.245850 5938 cpu_manager.go:414] "containerMap: removing container" podUID="80566eca-1f41-450c-968e-f917a76cc203" containerName="istio-proxy" Oct 30 09:50:49.245975 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.245855 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80566eca-1f41-450c-968e-f917a76cc203" containerName="istio-proxy" Oct 30 09:50:49.245975 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.245955 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb642f5e-4496-480d-a324-06c0df2fe3bb" containerName="payload-processing" Oct 30 09:50:49.245975 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.245968 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf5c897-be06-4cf9-9bdd-33974bd190e0" containerName="revoke-keys" Oct 30 09:50:49.245975 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.245976 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="80566eca-1f41-450c-968e-f917a76cc203" containerName="istio-proxy" Oct 30 09:50:49.251220 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.251195 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" Oct 30 09:50:49.267698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.267673 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz"] Oct 30 09:50:49.277625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.276438 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:50:49.277625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.276521 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:50:49.355991 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.355949 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kws5h\" (UniqueName: \"kubernetes.io/projected/96f40709-18d8-43c6-a11b-76d5098df362-kube-api-access-kws5h\") pod \"maas-api-revoke-sub-a88d7ff0-z6wtz\" (UID: \"96f40709-18d8-43c6-a11b-76d5098df362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" Oct 30 09:50:49.356155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.356090 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96f40709-18d8-43c6-a11b-76d5098df362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a88d7ff0-z6wtz\" (UID: \"96f40709-18d8-43c6-a11b-76d5098df362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" Oct 30 09:50:49.356199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.356154 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96f40709-18d8-43c6-a11b-76d5098df362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a88d7ff0-z6wtz\" (UID: \"96f40709-18d8-43c6-a11b-76d5098df362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" Oct 30 09:50:49.457013 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.456988 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kws5h\" (UniqueName: \"kubernetes.io/projected/96f40709-18d8-43c6-a11b-76d5098df362-kube-api-access-kws5h\") pod \"maas-api-revoke-sub-a88d7ff0-z6wtz\" (UID: \"96f40709-18d8-43c6-a11b-76d5098df362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" Oct 30 09:50:49.457158 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.457070 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96f40709-18d8-43c6-a11b-76d5098df362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a88d7ff0-z6wtz\" (UID: \"96f40709-18d8-43c6-a11b-76d5098df362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" Oct 30 09:50:49.457158 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.457112 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96f40709-18d8-43c6-a11b-76d5098df362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a88d7ff0-z6wtz\" (UID: \"96f40709-18d8-43c6-a11b-76d5098df362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" Oct 30 09:50:49.457743 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.457726 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96f40709-18d8-43c6-a11b-76d5098df362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a88d7ff0-z6wtz\" (UID: \"96f40709-18d8-43c6-a11b-76d5098df362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" Oct 30 09:50:49.459724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.459706 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96f40709-18d8-43c6-a11b-76d5098df362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a88d7ff0-z6wtz\" (UID: \"96f40709-18d8-43c6-a11b-76d5098df362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" Oct 30 09:50:49.476942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.476912 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kws5h\" (UniqueName: \"kubernetes.io/projected/96f40709-18d8-43c6-a11b-76d5098df362-kube-api-access-kws5h\") pod \"maas-api-revoke-sub-a88d7ff0-z6wtz\" (UID: \"96f40709-18d8-43c6-a11b-76d5098df362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" Oct 30 09:50:49.566521 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.566466 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" Oct 30 09:50:49.890292 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:49.890260 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz"] Oct 30 09:50:49.891008 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:50:49.890981 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f40709_18d8_43c6_a11b_76d5098df362.slice/crio-ea62a49018c773f8fc440ea51aadb031cc7e42ab03abe26dc7f8b87a7d14a653 WatchSource:0}: Error finding container ea62a49018c773f8fc440ea51aadb031cc7e42ab03abe26dc7f8b87a7d14a653: Status 404 returned error can't find the container with id ea62a49018c773f8fc440ea51aadb031cc7e42ab03abe26dc7f8b87a7d14a653 Oct 30 09:50:50.479062 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:50.478985 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" event={"ID":"96f40709-18d8-43c6-a11b-76d5098df362","Type":"ContainerStarted","Data":"5ea2241db578f7175a417121e5e6bfcc2c2b7b2f98b49430b07cfd7201671bf8"} Oct 30 09:50:50.479062 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:50.479042 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" event={"ID":"96f40709-18d8-43c6-a11b-76d5098df362","Type":"ContainerStarted","Data":"ea62a49018c773f8fc440ea51aadb031cc7e42ab03abe26dc7f8b87a7d14a653"} Oct 30 09:50:50.521164 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:50.521115 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" podStartSLOduration=1.5210985529999999 podStartE2EDuration="1.521098553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:50:50.519278904 +0000 UTC m=+3368.381984725" watchObservedRunningTime="2026-10-30 09:50:50.521098553 +0000 UTC m=+3368.383804375" Oct 30 09:50:51.488373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:51.488343 5938 generic.go:356] "Generic (PLEG): container finished" podID="96f40709-18d8-43c6-a11b-76d5098df362" containerID="5ea2241db578f7175a417121e5e6bfcc2c2b7b2f98b49430b07cfd7201671bf8" exitCode=0 Oct 30 09:50:51.488940 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:51.488374 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" event={"ID":"96f40709-18d8-43c6-a11b-76d5098df362","Type":"ContainerDied","Data":"5ea2241db578f7175a417121e5e6bfcc2c2b7b2f98b49430b07cfd7201671bf8"} Oct 30 09:50:52.668313 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:52.668288 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" Oct 30 09:50:52.697463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:52.697441 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96f40709-18d8-43c6-a11b-76d5098df362-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96f40709-18d8-43c6-a11b-76d5098df362-maas-api-xks-ca\") pod \"96f40709-18d8-43c6-a11b-76d5098df362\" (UID: \"96f40709-18d8-43c6-a11b-76d5098df362\") " Oct 30 09:50:52.697660 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:52.697619 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f40709-18d8-43c6-a11b-76d5098df362-kube-api-access-kws5h\" (UniqueName: \"kubernetes.io/projected/96f40709-18d8-43c6-a11b-76d5098df362-kube-api-access-kws5h\") pod \"96f40709-18d8-43c6-a11b-76d5098df362\" (UID: \"96f40709-18d8-43c6-a11b-76d5098df362\") " Oct 30 09:50:52.697729 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:52.697696 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96f40709-18d8-43c6-a11b-76d5098df362-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96f40709-18d8-43c6-a11b-76d5098df362-maas-api-openshift-ca\") pod \"96f40709-18d8-43c6-a11b-76d5098df362\" (UID: \"96f40709-18d8-43c6-a11b-76d5098df362\") " Oct 30 09:50:52.698062 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:52.698041 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f40709-18d8-43c6-a11b-76d5098df362-maas-api-openshift-ca" pod "96f40709-18d8-43c6-a11b-76d5098df362" (UID: "96f40709-18d8-43c6-a11b-76d5098df362"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:50:52.698264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:52.698246 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96f40709-18d8-43c6-a11b-76d5098df362-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:50:52.699647 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:52.699627 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96f40709-18d8-43c6-a11b-76d5098df362-maas-api-xks-ca" pod "96f40709-18d8-43c6-a11b-76d5098df362" (UID: "96f40709-18d8-43c6-a11b-76d5098df362"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:50:52.699911 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:52.699891 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f40709-18d8-43c6-a11b-76d5098df362-kube-api-access-kws5h" pod "96f40709-18d8-43c6-a11b-76d5098df362" (UID: "96f40709-18d8-43c6-a11b-76d5098df362"). InnerVolumeSpecName "kube-api-access-kws5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:50:52.799235 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:52.799152 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96f40709-18d8-43c6-a11b-76d5098df362-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:50:52.799235 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:52.799196 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kws5h\" (UniqueName: \"kubernetes.io/projected/96f40709-18d8-43c6-a11b-76d5098df362-kube-api-access-kws5h\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:50:53.503923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:53.503893 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" Oct 30 09:50:53.504107 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:53.503887 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz" event={"ID":"96f40709-18d8-43c6-a11b-76d5098df362","Type":"ContainerDied","Data":"ea62a49018c773f8fc440ea51aadb031cc7e42ab03abe26dc7f8b87a7d14a653"} Oct 30 09:50:53.504107 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:50:53.504023 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea62a49018c773f8fc440ea51aadb031cc7e42ab03abe26dc7f8b87a7d14a653" Oct 30 09:51:08.371835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:08.371794 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46"] Oct 30 09:51:08.375229 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:08.375162 5938 cpu_manager.go:414] "containerMap: removing container" podUID="96f40709-18d8-43c6-a11b-76d5098df362" containerName="revoke-keys" Oct 30 09:51:08.375406 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:08.375391 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96f40709-18d8-43c6-a11b-76d5098df362" containerName="revoke-keys" Oct 30 09:51:08.375770 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:08.375753 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="96f40709-18d8-43c6-a11b-76d5098df362" containerName="revoke-keys" Oct 30 09:51:08.469819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:08.469785 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46"] Oct 30 09:51:08.483308 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:08.483204 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" Oct 30 09:51:08.500732 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:08.500705 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-82ccfd-dockercfg-bwrq5\"" Oct 30 09:51:08.514451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:08.514425 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-82ccfd\"" Oct 30 09:51:08.649632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:08.649513 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf4c90b-e480-41a5-9522-045ee3d94955-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46\" (UID: \"9cf4c90b-e480-41a5-9522-045ee3d94955\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" Oct 30 09:51:08.649857 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:08.649830 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtszc\" (UniqueName: \"kubernetes.io/projected/9cf4c90b-e480-41a5-9522-045ee3d94955-kube-api-access-dtszc\") pod \"payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46\" (UID: \"9cf4c90b-e480-41a5-9522-045ee3d94955\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" Oct 30 09:51:08.750950 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:08.750922 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf4c90b-e480-41a5-9522-045ee3d94955-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46\" (UID: \"9cf4c90b-e480-41a5-9522-045ee3d94955\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" Oct 30 09:51:08.751130 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:08.751117 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtszc\" (UniqueName: \"kubernetes.io/projected/9cf4c90b-e480-41a5-9522-045ee3d94955-kube-api-access-dtszc\") pod \"payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46\" (UID: \"9cf4c90b-e480-41a5-9522-045ee3d94955\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" Oct 30 09:51:08.751656 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:08.751638 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf4c90b-e480-41a5-9522-045ee3d94955-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46\" (UID: \"9cf4c90b-e480-41a5-9522-045ee3d94955\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" Oct 30 09:51:08.771836 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:08.771811 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtszc\" (UniqueName: \"kubernetes.io/projected/9cf4c90b-e480-41a5-9522-045ee3d94955-kube-api-access-dtszc\") pod \"payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46\" (UID: \"9cf4c90b-e480-41a5-9522-045ee3d94955\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" Oct 30 09:51:08.807399 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:08.807374 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" Oct 30 09:51:09.115615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:09.115567 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46"] Oct 30 09:51:09.116092 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:51:09.116058 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf4c90b_e480_41a5_9522_045ee3d94955.slice/crio-fefbcf058e499a976a249c68732290d52e7ec5a56956bf4805100f4ea9cd3f33 WatchSource:0}: Error finding container fefbcf058e499a976a249c68732290d52e7ec5a56956bf4805100f4ea9cd3f33: Status 404 returned error can't find the container with id fefbcf058e499a976a249c68732290d52e7ec5a56956bf4805100f4ea9cd3f33 Oct 30 09:51:09.686772 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:09.686734 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" event={"ID":"9cf4c90b-e480-41a5-9522-045ee3d94955","Type":"ContainerStarted","Data":"d4e69608212cbb147031f247722a5b7891295e37e89616f15e32ea5dd3ca0fe6"} Oct 30 09:51:09.687191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:09.686794 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" event={"ID":"9cf4c90b-e480-41a5-9522-045ee3d94955","Type":"ContainerStarted","Data":"fefbcf058e499a976a249c68732290d52e7ec5a56956bf4805100f4ea9cd3f33"} Oct 30 09:51:09.687191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:09.686903 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" Oct 30 09:51:09.729178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:09.729135 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" podStartSLOduration=1.7291213619999999 podStartE2EDuration="1.729121362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:51:09.727018045 +0000 UTC m=+3387.589723878" watchObservedRunningTime="2026-10-30 09:51:09.729121362 +0000 UTC m=+3387.591827225" Oct 30 09:51:15.812108 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:51:15.812066 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:51:20.697314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:20.697261 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" Oct 30 09:51:21.031020 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:21.030934 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j"] Oct 30 09:51:21.034336 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:21.034307 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685ff309-4kt2j"] Oct 30 09:51:22.258388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:22.258353 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d415736-a914-4b4c-941c-0d2315ca7502" path="/var/lib/kubelet/pods/2d415736-a914-4b4c-941c-0d2315ca7502/volumes" Oct 30 09:51:25.848702 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:25.848667 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz"] Oct 30 09:51:25.853495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:25.853475 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:51:25.871120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:25.871095 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz"] Oct 30 09:51:25.880895 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:25.880872 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"agc-cred-568a5da76981\"" Oct 30 09:51:25.881067 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:25.880876 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"agc-cred-e771cb3dbc46\"" Oct 30 09:51:25.881067 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:25.880951 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"payload-processing-external-model-plugins-e2e-extmod-a-88cbc7\"" Oct 30 09:51:25.881269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:25.880970 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"kube-root-ca.crt\"" Oct 30 09:51:25.881269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:25.881112 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"payload-processing-external-model-e2e--185456a2-dockercfg-dfrl7\"" Oct 30 09:51:25.882793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:25.882771 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"routing-overlay\"" Oct 30 09:51:25.882903 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:25.882838 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"openshift-service-ca.crt\"" Oct 30 09:51:25.932008 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:25.931982 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7e4d501-83df-4eee-9008-a1da59cd2ee1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz\" (UID: \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:51:25.932167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:25.932142 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e4d501-83df-4eee-9008-a1da59cd2ee1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz\" (UID: \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:51:25.932297 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:25.932280 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c4b7\" (UniqueName: \"kubernetes.io/projected/f7e4d501-83df-4eee-9008-a1da59cd2ee1-kube-api-access-9c4b7\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz\" (UID: \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:51:25.932380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:25.932307 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7e4d501-83df-4eee-9008-a1da59cd2ee1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz\" (UID: \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:51:26.033659 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.033626 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c4b7\" (UniqueName: \"kubernetes.io/projected/f7e4d501-83df-4eee-9008-a1da59cd2ee1-kube-api-access-9c4b7\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz\" (UID: \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:51:26.033810 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.033671 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7e4d501-83df-4eee-9008-a1da59cd2ee1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz\" (UID: \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:51:26.033810 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.033777 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7e4d501-83df-4eee-9008-a1da59cd2ee1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz\" (UID: \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:51:26.033943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.033923 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e4d501-83df-4eee-9008-a1da59cd2ee1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz\" (UID: \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:51:26.034301 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.034280 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7e4d501-83df-4eee-9008-a1da59cd2ee1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz\" (UID: \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:51:26.034556 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.034538 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e4d501-83df-4eee-9008-a1da59cd2ee1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz\" (UID: \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:51:26.036517 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.036490 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7e4d501-83df-4eee-9008-a1da59cd2ee1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz\" (UID: \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:51:26.054251 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.054225 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c4b7\" (UniqueName: \"kubernetes.io/projected/f7e4d501-83df-4eee-9008-a1da59cd2ee1-kube-api-access-9c4b7\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz\" (UID: \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:51:26.169618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.169240 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:51:26.502453 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.502405 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz"] Oct 30 09:51:26.503864 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:51:26.503832 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e4d501_83df_4eee_9008_a1da59cd2ee1.slice/crio-6258845e8f23134d33e95a5c12a0dc6627df8768598cab88785f414354ce955b WatchSource:0}: Error finding container 6258845e8f23134d33e95a5c12a0dc6627df8768598cab88785f414354ce955b: Status 404 returned error can't find the container with id 6258845e8f23134d33e95a5c12a0dc6627df8768598cab88785f414354ce955b Oct 30 09:51:26.855199 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.855125 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl"] Oct 30 09:51:26.859642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.859620 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:51:26.876407 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.876368 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d66d22\"/\"agc-cred-f732860533e5\"" Oct 30 09:51:26.881620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.881580 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl"] Oct 30 09:51:26.885974 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.885948 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" event={"ID":"f7e4d501-83df-4eee-9008-a1da59cd2ee1","Type":"ContainerStarted","Data":"45af0ca963909d595ab1d6d2b2c8aa45a221eb58d6ac5e5eef9d5e310f0f312b"} Oct 30 09:51:26.886120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.885991 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" event={"ID":"f7e4d501-83df-4eee-9008-a1da59cd2ee1","Type":"ContainerStarted","Data":"6258845e8f23134d33e95a5c12a0dc6627df8768598cab88785f414354ce955b"} Oct 30 09:51:26.886120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.886029 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:51:26.945845 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.945796 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" podStartSLOduration=1.945782106 podStartE2EDuration="1.945782106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:51:26.944616477 +0000 UTC m=+3404.807322298" watchObservedRunningTime="2026-10-30 09:51:26.945782106 +0000 UTC m=+3404.808487926" Oct 30 09:51:26.949766 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.949737 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28fjp\" (UniqueName: \"kubernetes.io/projected/befb486a-0145-4d7e-934b-6d4d01e5aa0f-kube-api-access-28fjp\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl\" (UID: \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:51:26.949965 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.949943 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/befb486a-0145-4d7e-934b-6d4d01e5aa0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl\" (UID: \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:51:26.950019 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.949983 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/befb486a-0145-4d7e-934b-6d4d01e5aa0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl\" (UID: \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:51:26.950067 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:26.950053 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/befb486a-0145-4d7e-934b-6d4d01e5aa0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl\" (UID: \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:51:27.050935 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:27.050896 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28fjp\" (UniqueName: \"kubernetes.io/projected/befb486a-0145-4d7e-934b-6d4d01e5aa0f-kube-api-access-28fjp\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl\" (UID: \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:51:27.051150 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:27.051131 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/befb486a-0145-4d7e-934b-6d4d01e5aa0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl\" (UID: \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:51:27.051239 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:27.051181 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/befb486a-0145-4d7e-934b-6d4d01e5aa0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl\" (UID: \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:51:27.051320 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:27.051244 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/befb486a-0145-4d7e-934b-6d4d01e5aa0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl\" (UID: \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:51:27.051950 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:27.051922 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/befb486a-0145-4d7e-934b-6d4d01e5aa0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl\" (UID: \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:51:27.052086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:27.052066 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/befb486a-0145-4d7e-934b-6d4d01e5aa0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl\" (UID: \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:51:27.053890 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:27.053865 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/befb486a-0145-4d7e-934b-6d4d01e5aa0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl\" (UID: \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:51:27.071882 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:27.071862 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28fjp\" (UniqueName: \"kubernetes.io/projected/befb486a-0145-4d7e-934b-6d4d01e5aa0f-kube-api-access-28fjp\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl\" (UID: \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:51:27.180937 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:27.180909 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:51:27.483725 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:27.483700 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl"] Oct 30 09:51:27.484581 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:51:27.484557 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbefb486a_0145_4d7e_934b_6d4d01e5aa0f.slice/crio-562ae5f1adde7deb1291cd368ab57b3dcde39ebaeabe23608838223b2abc973f WatchSource:0}: Error finding container 562ae5f1adde7deb1291cd368ab57b3dcde39ebaeabe23608838223b2abc973f: Status 404 returned error can't find the container with id 562ae5f1adde7deb1291cd368ab57b3dcde39ebaeabe23608838223b2abc973f Oct 30 09:51:27.895124 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:27.895084 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" event={"ID":"befb486a-0145-4d7e-934b-6d4d01e5aa0f","Type":"ContainerStarted","Data":"f6723b818a6e24a4bc5b7e27dfc7a26304f629536842bdaef33fe5758bec8a47"} Oct 30 09:51:27.895535 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:27.895147 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:51:27.895535 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:27.895165 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" event={"ID":"befb486a-0145-4d7e-934b-6d4d01e5aa0f","Type":"ContainerStarted","Data":"562ae5f1adde7deb1291cd368ab57b3dcde39ebaeabe23608838223b2abc973f"} Oct 30 09:51:27.938513 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:27.938464 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" podStartSLOduration=1.938449212 podStartE2EDuration="1.938449212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:51:27.935385928 +0000 UTC m=+3405.798091752" watchObservedRunningTime="2026-10-30 09:51:27.938449212 +0000 UTC m=+3405.801155054" Oct 30 09:51:31.030800 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:31.030771 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t"] Oct 30 09:51:31.043018 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:31.042987 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e956a3-w282t"] Oct 30 09:51:32.258205 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:32.258173 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2f279e-75fd-424e-8e3f-edd77de0d4a4" path="/var/lib/kubelet/pods/7d2f279e-75fd-424e-8e3f-edd77de0d4a4/volumes" Oct 30 09:51:33.486797 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:33.486763 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx"] Oct 30 09:51:37.898200 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:37.898170 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:51:38.487719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.487683 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" podUID="98ce89f9-6fe0-47a1-867e-40806e5aa6fb" containerName="payload-processing" containerID="cri-o://1cd7eb8c13d95fc3691cd9581bd733d6a51568de9235d870b97a987ce6932e20" gracePeriod=25 Oct 30 09:51:38.770741 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.770717 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:51:38.815906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.815876 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-plugins-config-volume\") pod \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\" (UID: \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\") " Oct 30 09:51:38.816698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.815938 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-provider-credentials\") pod \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\" (UID: \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\") " Oct 30 09:51:38.816698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.816157 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-kube-api-access-pgn6x\" (UniqueName: \"kubernetes.io/projected/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-kube-api-access-pgn6x\") pod \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\" (UID: \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\") " Oct 30 09:51:38.816698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.816252 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-routing-overlay-volume\") pod \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\" (UID: \"98ce89f9-6fe0-47a1-867e-40806e5aa6fb\") " Oct 30 09:51:38.816698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.816339 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-plugins-config-volume" pod "98ce89f9-6fe0-47a1-867e-40806e5aa6fb" (UID: "98ce89f9-6fe0-47a1-867e-40806e5aa6fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:51:38.817016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.816803 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:51:38.817016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.816838 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-routing-overlay-volume" pod "98ce89f9-6fe0-47a1-867e-40806e5aa6fb" (UID: "98ce89f9-6fe0-47a1-867e-40806e5aa6fb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:51:38.818677 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.818651 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-provider-credentials" pod "98ce89f9-6fe0-47a1-867e-40806e5aa6fb" (UID: "98ce89f9-6fe0-47a1-867e-40806e5aa6fb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:51:38.818868 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.818846 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-kube-api-access-pgn6x" pod "98ce89f9-6fe0-47a1-867e-40806e5aa6fb" (UID: "98ce89f9-6fe0-47a1-867e-40806e5aa6fb"). InnerVolumeSpecName "kube-api-access-pgn6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:51:38.906390 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.906365 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:51:38.917609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.917563 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:51:38.917705 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.917623 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgn6x\" (UniqueName: \"kubernetes.io/projected/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-kube-api-access-pgn6x\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:51:38.917705 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.917636 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98ce89f9-6fe0-47a1-867e-40806e5aa6fb-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:51:38.979450 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.978769 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6"] Oct 30 09:51:38.997633 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.997530 5938 generic.go:356] "Generic (PLEG): container finished" podID="98ce89f9-6fe0-47a1-867e-40806e5aa6fb" containerID="1cd7eb8c13d95fc3691cd9581bd733d6a51568de9235d870b97a987ce6932e20" exitCode=0 Oct 30 09:51:38.997633 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.997620 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" Oct 30 09:51:38.997871 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.997738 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" event={"ID":"98ce89f9-6fe0-47a1-867e-40806e5aa6fb","Type":"ContainerDied","Data":"1cd7eb8c13d95fc3691cd9581bd733d6a51568de9235d870b97a987ce6932e20"} Oct 30 09:51:38.997871 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.997787 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx" event={"ID":"98ce89f9-6fe0-47a1-867e-40806e5aa6fb","Type":"ContainerDied","Data":"513be8af0b82f6b0a3177a93771436c49544177782a26767a33f528f0da07551"} Oct 30 09:51:38.997871 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:38.997816 5938 scope.go:122] "RemoveContainer" containerID="1cd7eb8c13d95fc3691cd9581bd733d6a51568de9235d870b97a987ce6932e20" Oct 30 09:51:39.012008 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:39.011686 5938 scope.go:122] "RemoveContainer" containerID="1cd7eb8c13d95fc3691cd9581bd733d6a51568de9235d870b97a987ce6932e20" Oct 30 09:51:39.012362 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:51:39.012341 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd7eb8c13d95fc3691cd9581bd733d6a51568de9235d870b97a987ce6932e20\": container with ID starting with 1cd7eb8c13d95fc3691cd9581bd733d6a51568de9235d870b97a987ce6932e20 not found: ID does not exist" containerID="1cd7eb8c13d95fc3691cd9581bd733d6a51568de9235d870b97a987ce6932e20" Oct 30 09:51:39.012443 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:39.012374 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd7eb8c13d95fc3691cd9581bd733d6a51568de9235d870b97a987ce6932e20"} err="failed to get container status \"1cd7eb8c13d95fc3691cd9581bd733d6a51568de9235d870b97a987ce6932e20\": rpc error: code = NotFound desc = could not find container \"1cd7eb8c13d95fc3691cd9581bd733d6a51568de9235d870b97a987ce6932e20\": container with ID starting with 1cd7eb8c13d95fc3691cd9581bd733d6a51568de9235d870b97a987ce6932e20 not found: ID does not exist" Oct 30 09:51:39.065075 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:39.065036 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx"] Oct 30 09:51:39.076020 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:39.075992 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546b5xllx"] Oct 30 09:51:40.258395 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:40.258361 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ce89f9-6fe0-47a1-867e-40806e5aa6fb" path="/var/lib/kubelet/pods/98ce89f9-6fe0-47a1-867e-40806e5aa6fb/volumes" Oct 30 09:51:43.979145 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:43.979103 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" podUID="f935fb8b-e17c-452c-a830-279e0f52ad1e" containerName="payload-processing" containerID="cri-o://c454d642818b1da422cd7aa82bf1e3a8117f624186b32d419fb231ef537b2110" gracePeriod=25 Oct 30 09:51:44.028030 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.028007 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv"] Oct 30 09:51:44.033869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.033812 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00702f57-tv6rv"] Oct 30 09:51:44.210442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.210414 5938 scope.go:122] "RemoveContainer" containerID="bde2c52a8bae69afd7dff15efe5fbca351274e76370a665397fb6c01431c8054" Oct 30 09:51:44.260165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.260131 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aeaf185-a46f-4b3c-a52d-8c97f4733dd6" path="/var/lib/kubelet/pods/9aeaf185-a46f-4b3c-a52d-8c97f4733dd6/volumes" Oct 30 09:51:44.267300 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.267274 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:51:44.383852 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.383824 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f935fb8b-e17c-452c-a830-279e0f52ad1e-kube-api-access-67kb8\" (UniqueName: \"kubernetes.io/projected/f935fb8b-e17c-452c-a830-279e0f52ad1e-kube-api-access-67kb8\") pod \"f935fb8b-e17c-452c-a830-279e0f52ad1e\" (UID: \"f935fb8b-e17c-452c-a830-279e0f52ad1e\") " Oct 30 09:51:44.384021 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.384007 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f935fb8b-e17c-452c-a830-279e0f52ad1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f935fb8b-e17c-452c-a830-279e0f52ad1e-plugins-config-volume\") pod \"f935fb8b-e17c-452c-a830-279e0f52ad1e\" (UID: \"f935fb8b-e17c-452c-a830-279e0f52ad1e\") " Oct 30 09:51:44.384080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.384069 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f935fb8b-e17c-452c-a830-279e0f52ad1e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f935fb8b-e17c-452c-a830-279e0f52ad1e-routing-overlay-volume\") pod \"f935fb8b-e17c-452c-a830-279e0f52ad1e\" (UID: \"f935fb8b-e17c-452c-a830-279e0f52ad1e\") " Oct 30 09:51:44.384118 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.384101 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f935fb8b-e17c-452c-a830-279e0f52ad1e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f935fb8b-e17c-452c-a830-279e0f52ad1e-provider-credentials\") pod \"f935fb8b-e17c-452c-a830-279e0f52ad1e\" (UID: \"f935fb8b-e17c-452c-a830-279e0f52ad1e\") " Oct 30 09:51:44.384389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.384366 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f935fb8b-e17c-452c-a830-279e0f52ad1e-plugins-config-volume" pod "f935fb8b-e17c-452c-a830-279e0f52ad1e" (UID: "f935fb8b-e17c-452c-a830-279e0f52ad1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:51:44.384880 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.384454 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f935fb8b-e17c-452c-a830-279e0f52ad1e-routing-overlay-volume" pod "f935fb8b-e17c-452c-a830-279e0f52ad1e" (UID: "f935fb8b-e17c-452c-a830-279e0f52ad1e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:51:44.384880 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.384708 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f935fb8b-e17c-452c-a830-279e0f52ad1e-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:51:44.384880 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.384733 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f935fb8b-e17c-452c-a830-279e0f52ad1e-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:51:44.386435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.386403 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f935fb8b-e17c-452c-a830-279e0f52ad1e-kube-api-access-67kb8" pod "f935fb8b-e17c-452c-a830-279e0f52ad1e" (UID: "f935fb8b-e17c-452c-a830-279e0f52ad1e"). InnerVolumeSpecName "kube-api-access-67kb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:51:44.386435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.386406 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f935fb8b-e17c-452c-a830-279e0f52ad1e-provider-credentials" pod "f935fb8b-e17c-452c-a830-279e0f52ad1e" (UID: "f935fb8b-e17c-452c-a830-279e0f52ad1e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:51:44.486341 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.486269 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f935fb8b-e17c-452c-a830-279e0f52ad1e-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:51:44.486341 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:44.486296 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67kb8\" (UniqueName: \"kubernetes.io/projected/f935fb8b-e17c-452c-a830-279e0f52ad1e-kube-api-access-67kb8\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:51:45.050666 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:45.050636 5938 generic.go:356] "Generic (PLEG): container finished" podID="f935fb8b-e17c-452c-a830-279e0f52ad1e" containerID="c454d642818b1da422cd7aa82bf1e3a8117f624186b32d419fb231ef537b2110" exitCode=0 Oct 30 09:51:45.051102 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:45.050685 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" Oct 30 09:51:45.051102 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:45.050704 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" event={"ID":"f935fb8b-e17c-452c-a830-279e0f52ad1e","Type":"ContainerDied","Data":"c454d642818b1da422cd7aa82bf1e3a8117f624186b32d419fb231ef537b2110"} Oct 30 09:51:45.051102 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:45.050745 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6" event={"ID":"f935fb8b-e17c-452c-a830-279e0f52ad1e","Type":"ContainerDied","Data":"6353e3a186d8549b57d4ce2c3314def469ff5733a49847441911f4b5acc5fc54"} Oct 30 09:51:45.051102 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:45.050765 5938 scope.go:122] "RemoveContainer" containerID="c454d642818b1da422cd7aa82bf1e3a8117f624186b32d419fb231ef537b2110" Oct 30 09:51:45.062250 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:45.062235 5938 scope.go:122] "RemoveContainer" containerID="c454d642818b1da422cd7aa82bf1e3a8117f624186b32d419fb231ef537b2110" Oct 30 09:51:45.062502 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:51:45.062486 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c454d642818b1da422cd7aa82bf1e3a8117f624186b32d419fb231ef537b2110\": container with ID starting with c454d642818b1da422cd7aa82bf1e3a8117f624186b32d419fb231ef537b2110 not found: ID does not exist" containerID="c454d642818b1da422cd7aa82bf1e3a8117f624186b32d419fb231ef537b2110" Oct 30 09:51:45.062555 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:45.062509 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c454d642818b1da422cd7aa82bf1e3a8117f624186b32d419fb231ef537b2110"} err="failed to get container status \"c454d642818b1da422cd7aa82bf1e3a8117f624186b32d419fb231ef537b2110\": rpc error: code = NotFound desc = could not find container \"c454d642818b1da422cd7aa82bf1e3a8117f624186b32d419fb231ef537b2110\": container with ID starting with c454d642818b1da422cd7aa82bf1e3a8117f624186b32d419fb231ef537b2110 not found: ID does not exist" Oct 30 09:51:45.119253 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:45.119224 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6"] Oct 30 09:51:45.125643 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:45.125613 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6cgt2n6"] Oct 30 09:51:45.808801 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:51:45.808763 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:51:46.258198 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:51:46.258162 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f935fb8b-e17c-452c-a830-279e0f52ad1e" path="/var/lib/kubelet/pods/f935fb8b-e17c-452c-a830-279e0f52ad1e/volumes" Oct 30 09:52:15.809051 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:52:15.808974 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:52:32.027414 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:32.027372 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9"] Oct 30 09:52:32.040072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:32.040042 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1324ac1-vkqt9"] Oct 30 09:52:32.259023 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:32.258991 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72" path="/var/lib/kubelet/pods/6a0b2a53-fdba-4dfd-a4e4-65acb7f4fe72/volumes" Oct 30 09:52:33.188046 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.188016 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm"] Oct 30 09:52:33.189004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.188975 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f935fb8b-e17c-452c-a830-279e0f52ad1e" containerName="payload-processing" Oct 30 09:52:33.189004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.189002 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f935fb8b-e17c-452c-a830-279e0f52ad1e" containerName="payload-processing" Oct 30 09:52:33.189239 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.189042 5938 cpu_manager.go:414] "containerMap: removing container" podUID="98ce89f9-6fe0-47a1-867e-40806e5aa6fb" containerName="payload-processing" Oct 30 09:52:33.189239 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.189048 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ce89f9-6fe0-47a1-867e-40806e5aa6fb" containerName="payload-processing" Oct 30 09:52:33.189239 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.189181 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f935fb8b-e17c-452c-a830-279e0f52ad1e" containerName="payload-processing" Oct 30 09:52:33.189239 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.189198 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ce89f9-6fe0-47a1-867e-40806e5aa6fb" containerName="payload-processing" Oct 30 09:52:33.193253 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.193235 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.211773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.211749 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm"] Oct 30 09:52:33.220008 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.219983 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bdd208\"/\"default-dockercfg-ns5w9\"" Oct 30 09:52:33.220133 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.220095 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bdd208\"/\"kube-root-ca.crt\"" Oct 30 09:52:33.223142 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.223122 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bdd208\"/\"test-model-bdd208-kserve-self-signed-certs\"" Oct 30 09:52:33.223330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.223310 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bdd208\"/\"openshift-service-ca.crt\"" Oct 30 09:52:33.327562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.327532 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2flms\" (UniqueName: \"kubernetes.io/projected/89b3980d-9c68-495b-a417-ca06457f2ea9-kube-api-access-2flms\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.327805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.327687 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-home\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.327805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.327732 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-dshm\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.327947 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.327867 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-tmp-dir\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.327947 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.327915 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-model-cache\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.328078 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.327961 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b3980d-9c68-495b-a417-ca06457f2ea9-tls-certs\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.441278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.440175 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-tmp-dir\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.441278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.440228 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-model-cache\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.441278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.440307 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b3980d-9c68-495b-a417-ca06457f2ea9-tls-certs\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.441278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.440638 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2flms\" (UniqueName: \"kubernetes.io/projected/89b3980d-9c68-495b-a417-ca06457f2ea9-kube-api-access-2flms\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.441278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.440723 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-home\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.441278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.440785 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-dshm\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.442016 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.441990 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-home\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.449363 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.446467 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-tmp-dir\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.453672 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.449999 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-model-cache\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.466189 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.466160 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2flms\" (UniqueName: \"kubernetes.io/projected/89b3980d-9c68-495b-a417-ca06457f2ea9-kube-api-access-2flms\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.476629 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.476574 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-dshm\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.481176 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.481142 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b3980d-9c68-495b-a417-ca06457f2ea9-tls-certs\") pod \"test-model-bdd208-kserve-f7ffc85f9-b6lcm\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.515651 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.512895 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:33.879892 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:33.879866 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm"] Oct 30 09:52:33.880578 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:52:33.880553 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89b3980d_9c68_495b_a417_ca06457f2ea9.slice/crio-a9854053d3f4853839d160f37568402f1cfa582dd2526be6464d089cbc51fb80 WatchSource:0}: Error finding container a9854053d3f4853839d160f37568402f1cfa582dd2526be6464d089cbc51fb80: Status 404 returned error can't find the container with id a9854053d3f4853839d160f37568402f1cfa582dd2526be6464d089cbc51fb80 Oct 30 09:52:34.447943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:34.447906 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" event={"ID":"89b3980d-9c68-495b-a417-ca06457f2ea9","Type":"ContainerStarted","Data":"c9fe4791a5cfd7ad14c2e426c1782b8327dda61f375dcb708b7a12d1133522a6"} Oct 30 09:52:34.447943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:34.447957 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" event={"ID":"89b3980d-9c68-495b-a417-ca06457f2ea9","Type":"ContainerStarted","Data":"a9854053d3f4853839d160f37568402f1cfa582dd2526be6464d089cbc51fb80"} Oct 30 09:52:34.493607 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:34.493547 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" podStartSLOduration=1.493533927 podStartE2EDuration="1.493533927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:52:34.490479278 +0000 UTC m=+3472.353185100" watchObservedRunningTime="2026-10-30 09:52:34.493533927 +0000 UTC m=+3472.356239748" Oct 30 09:52:34.513734 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:34.513707 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:43.027812 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:43.027779 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d"] Oct 30 09:52:43.040756 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:43.040724 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07fb049-2tf8d"] Oct 30 09:52:43.514249 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:43.514212 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:43.528110 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:43.528087 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:44.245464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:44.245436 5938 scope.go:122] "RemoveContainer" containerID="baac80030ec2bc55a14ecf37a413d2062f175761bb3e1cbeee2c602a1c7bd585" Oct 30 09:52:44.261575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:44.261552 5938 scope.go:122] "RemoveContainer" containerID="38269781954c7400108f02c6a1898614e8ad73f7ec06689a12782bfd55dc3e2b" Oct 30 09:52:44.263459 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:44.263434 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e17ac2d2-e63e-4959-951d-d8bc30d14303" path="/var/lib/kubelet/pods/e17ac2d2-e63e-4959-951d-d8bc30d14303/volumes" Oct 30 09:52:44.273995 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:44.273970 5938 scope.go:122] "RemoveContainer" containerID="b478fada56726373ae3ce95dee061d62f37615881490d26b1da123bd043efc55" Oct 30 09:52:44.539610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:44.539510 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:52:45.809683 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:52:45.809647 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:52:48.194844 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.194811 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz"] Oct 30 09:52:48.202246 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.202225 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.222023 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.221997 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz"] Oct 30 09:52:48.231663 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.231636 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-82ccfd-openshift-default-dockercfg-rlbdn\"" Oct 30 09:52:48.271262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.271232 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jlc2\" (UniqueName: \"kubernetes.io/projected/c7325b5f-f90d-44a9-876d-48245c0fe2b0-kube-api-access-2jlc2\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.271450 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.271279 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-token\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.271450 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.271414 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-workload-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.271559 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.271472 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-data\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.271634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.271566 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-workload-certs\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.271704 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.271691 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-credential-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.271766 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.271736 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istiod-ca-cert\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.271856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.271839 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-envoy\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.271910 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.271892 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-podinfo\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.285842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.285817 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds"] Oct 30 09:52:48.290549 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.290533 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" Oct 30 09:52:48.309312 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.309293 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds"] Oct 30 09:52:48.321863 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.321840 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:52:48.322065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.322046 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:52:48.373407 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.373380 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-token\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.373544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.373457 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-workload-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.373544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.373503 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-data\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.373701 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.373560 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-workload-certs\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.373701 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.373647 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a46a6f-5252-4a98-85be-982321aab480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67b4d96e-h4kds\" (UID: \"66a46a6f-5252-4a98-85be-982321aab480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" Oct 30 09:52:48.373833 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.373702 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a46a6f-5252-4a98-85be-982321aab480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67b4d96e-h4kds\" (UID: \"66a46a6f-5252-4a98-85be-982321aab480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" Oct 30 09:52:48.373833 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.373762 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-credential-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.373948 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.373866 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istiod-ca-cert\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.373948 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.373913 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58h97\" (UniqueName: \"kubernetes.io/projected/66a46a6f-5252-4a98-85be-982321aab480-kube-api-access-58h97\") pod \"maas-api-revoke-sub-67b4d96e-h4kds\" (UID: \"66a46a6f-5252-4a98-85be-982321aab480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" Oct 30 09:52:48.373948 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.373937 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-workload-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.374111 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.373956 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-data\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.374111 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.373994 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-envoy\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.374111 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.374017 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-workload-certs\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.374111 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.374026 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-podinfo\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.374345 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.374109 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-credential-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.374345 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.374118 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jlc2\" (UniqueName: \"kubernetes.io/projected/c7325b5f-f90d-44a9-876d-48245c0fe2b0-kube-api-access-2jlc2\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.374629 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.374569 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istiod-ca-cert\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.376473 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.376450 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-envoy\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.376575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.376559 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-podinfo\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.401438 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.401414 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-token\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.402304 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.402282 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jlc2\" (UniqueName: \"kubernetes.io/projected/c7325b5f-f90d-44a9-876d-48245c0fe2b0-kube-api-access-2jlc2\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.476006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.475922 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a46a6f-5252-4a98-85be-982321aab480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67b4d96e-h4kds\" (UID: \"66a46a6f-5252-4a98-85be-982321aab480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" Oct 30 09:52:48.476006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.475992 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a46a6f-5252-4a98-85be-982321aab480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67b4d96e-h4kds\" (UID: \"66a46a6f-5252-4a98-85be-982321aab480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" Oct 30 09:52:48.476207 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.476096 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58h97\" (UniqueName: \"kubernetes.io/projected/66a46a6f-5252-4a98-85be-982321aab480-kube-api-access-58h97\") pod \"maas-api-revoke-sub-67b4d96e-h4kds\" (UID: \"66a46a6f-5252-4a98-85be-982321aab480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" Oct 30 09:52:48.476687 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.476668 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a46a6f-5252-4a98-85be-982321aab480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67b4d96e-h4kds\" (UID: \"66a46a6f-5252-4a98-85be-982321aab480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" Oct 30 09:52:48.478264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.478245 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a46a6f-5252-4a98-85be-982321aab480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67b4d96e-h4kds\" (UID: \"66a46a6f-5252-4a98-85be-982321aab480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" Oct 30 09:52:48.496467 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.496445 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58h97\" (UniqueName: \"kubernetes.io/projected/66a46a6f-5252-4a98-85be-982321aab480-kube-api-access-58h97\") pod \"maas-api-revoke-sub-67b4d96e-h4kds\" (UID: \"66a46a6f-5252-4a98-85be-982321aab480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" Oct 30 09:52:48.517553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.517512 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:48.604848 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.604819 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" Oct 30 09:52:48.835412 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.835338 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz"] Oct 30 09:52:48.840031 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:52:48.840005 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7325b5f_f90d_44a9_876d_48245c0fe2b0.slice/crio-d6576915d06f3b4b35e5c49c2c89133b4c5845a553cdbc99c1bddeb89be42ba4 WatchSource:0}: Error finding container d6576915d06f3b4b35e5c49c2c89133b4c5845a553cdbc99c1bddeb89be42ba4: Status 404 returned error can't find the container with id d6576915d06f3b4b35e5c49c2c89133b4c5845a553cdbc99c1bddeb89be42ba4 Oct 30 09:52:48.843034 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.842999 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:52:48.843134 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.843066 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:52:48.843134 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.843097 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:52:48.928250 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:48.928217 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds"] Oct 30 09:52:48.929698 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:52:48.929669 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a46a6f_5252_4a98_85be_982321aab480.slice/crio-058255582e3ab02e28b20382d6f3ad85025020c9b944a3dcec05f1b2e9f3b168 WatchSource:0}: Error finding container 058255582e3ab02e28b20382d6f3ad85025020c9b944a3dcec05f1b2e9f3b168: Status 404 returned error can't find the container with id 058255582e3ab02e28b20382d6f3ad85025020c9b944a3dcec05f1b2e9f3b168 Oct 30 09:52:49.577824 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:49.577727 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" event={"ID":"c7325b5f-f90d-44a9-876d-48245c0fe2b0","Type":"ContainerStarted","Data":"3c14ec9b0900dc0c0ffefa848b0f8f51164cf74a8c85c28d922376f27aebc8f5"} Oct 30 09:52:49.577824 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:49.577783 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" event={"ID":"c7325b5f-f90d-44a9-876d-48245c0fe2b0","Type":"ContainerStarted","Data":"d6576915d06f3b4b35e5c49c2c89133b4c5845a553cdbc99c1bddeb89be42ba4"} Oct 30 09:52:49.579513 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:49.579481 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" event={"ID":"66a46a6f-5252-4a98-85be-982321aab480","Type":"ContainerStarted","Data":"032374664dd0f4f9078b2c6dc9cd2adae1f58165b630a4b3b9e424f514376960"} Oct 30 09:52:49.579694 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:49.579537 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" event={"ID":"66a46a6f-5252-4a98-85be-982321aab480","Type":"ContainerStarted","Data":"058255582e3ab02e28b20382d6f3ad85025020c9b944a3dcec05f1b2e9f3b168"} Oct 30 09:52:49.638575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:49.638523 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" podStartSLOduration=1.638507038 podStartE2EDuration="1.638507038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:52:49.621660319 +0000 UTC m=+3487.484366144" watchObservedRunningTime="2026-10-30 09:52:49.638507038 +0000 UTC m=+3487.501212925" Oct 30 09:52:49.662416 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:49.662363 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" podStartSLOduration=1.662349786 podStartE2EDuration="1.662349786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:52:49.661342134 +0000 UTC m=+3487.524047967" watchObservedRunningTime="2026-10-30 09:52:49.662349786 +0000 UTC m=+3487.525055607" Oct 30 09:52:50.347057 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.346566 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l"] Oct 30 09:52:50.352918 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.352891 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" Oct 30 09:52:50.368096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.368064 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l"] Oct 30 09:52:50.369808 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.369786 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a479df\"" Oct 30 09:52:50.401065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.401035 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a479df-dockercfg-vtvk4\"" Oct 30 09:52:50.457098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.457060 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s"] Oct 30 09:52:50.462924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.462898 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" Oct 30 09:52:50.516610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.511693 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7df54\" (UniqueName: \"kubernetes.io/projected/e9c527fc-edd4-4a04-a781-483921b54c6b-kube-api-access-7df54\") pod \"payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l\" (UID: \"e9c527fc-edd4-4a04-a781-483921b54c6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" Oct 30 09:52:50.527617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.520278 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:50.528081 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.527978 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c527fc-edd4-4a04-a781-483921b54c6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l\" (UID: \"e9c527fc-edd4-4a04-a781-483921b54c6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" Oct 30 09:52:50.584638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.576688 5938 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 09:52:50.584638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.576747 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" podUID="c7325b5f-f90d-44a9-876d-48245c0fe2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 09:52:50.584638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.579204 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s"] Oct 30 09:52:50.598041 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.597963 5938 generic.go:356] "Generic (PLEG): container finished" podID="66a46a6f-5252-4a98-85be-982321aab480" containerID="032374664dd0f4f9078b2c6dc9cd2adae1f58165b630a4b3b9e424f514376960" exitCode=0 Oct 30 09:52:50.598672 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.598608 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" event={"ID":"66a46a6f-5252-4a98-85be-982321aab480","Type":"ContainerDied","Data":"032374664dd0f4f9078b2c6dc9cd2adae1f58165b630a4b3b9e424f514376960"} Oct 30 09:52:50.676129 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.676092 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c527fc-edd4-4a04-a781-483921b54c6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l\" (UID: \"e9c527fc-edd4-4a04-a781-483921b54c6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" Oct 30 09:52:50.677635 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.677605 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80bc4ca3-dd13-4fa4-900f-82b93b520ffe-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s\" (UID: \"80bc4ca3-dd13-4fa4-900f-82b93b520ffe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" Oct 30 09:52:50.678114 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.678092 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtlm4\" (UniqueName: \"kubernetes.io/projected/80bc4ca3-dd13-4fa4-900f-82b93b520ffe-kube-api-access-qtlm4\") pod \"payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s\" (UID: \"80bc4ca3-dd13-4fa4-900f-82b93b520ffe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" Oct 30 09:52:50.678885 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.678864 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7df54\" (UniqueName: \"kubernetes.io/projected/e9c527fc-edd4-4a04-a781-483921b54c6b-kube-api-access-7df54\") pod \"payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l\" (UID: \"e9c527fc-edd4-4a04-a781-483921b54c6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" Oct 30 09:52:50.680146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.677461 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c527fc-edd4-4a04-a781-483921b54c6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l\" (UID: \"e9c527fc-edd4-4a04-a781-483921b54c6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" Oct 30 09:52:50.700647 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.700565 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7df54\" (UniqueName: \"kubernetes.io/projected/e9c527fc-edd4-4a04-a781-483921b54c6b-kube-api-access-7df54\") pod \"payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l\" (UID: \"e9c527fc-edd4-4a04-a781-483921b54c6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" Oct 30 09:52:50.781141 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.781103 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtlm4\" (UniqueName: \"kubernetes.io/projected/80bc4ca3-dd13-4fa4-900f-82b93b520ffe-kube-api-access-qtlm4\") pod \"payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s\" (UID: \"80bc4ca3-dd13-4fa4-900f-82b93b520ffe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" Oct 30 09:52:50.781323 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.781296 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80bc4ca3-dd13-4fa4-900f-82b93b520ffe-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s\" (UID: \"80bc4ca3-dd13-4fa4-900f-82b93b520ffe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" Oct 30 09:52:50.782254 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.782225 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80bc4ca3-dd13-4fa4-900f-82b93b520ffe-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s\" (UID: \"80bc4ca3-dd13-4fa4-900f-82b93b520ffe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" Oct 30 09:52:50.802269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.802235 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtlm4\" (UniqueName: \"kubernetes.io/projected/80bc4ca3-dd13-4fa4-900f-82b93b520ffe-kube-api-access-qtlm4\") pod \"payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s\" (UID: \"80bc4ca3-dd13-4fa4-900f-82b93b520ffe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" Oct 30 09:52:50.971362 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:50.971319 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" Oct 30 09:52:51.096227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:51.096194 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" Oct 30 09:52:51.301796 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:51.301763 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l"] Oct 30 09:52:51.450936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:51.450827 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s"] Oct 30 09:52:51.456745 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:52:51.456712 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80bc4ca3_dd13_4fa4_900f_82b93b520ffe.slice/crio-c1db1804633c6f38bdfa8bc3c417a7a72c768225da6c264728327818556c62a1 WatchSource:0}: Error finding container c1db1804633c6f38bdfa8bc3c417a7a72c768225da6c264728327818556c62a1: Status 404 returned error can't find the container with id c1db1804633c6f38bdfa8bc3c417a7a72c768225da6c264728327818556c62a1 Oct 30 09:52:51.525750 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:51.525720 5938 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 09:52:51.525986 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:51.525957 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" podUID="c7325b5f-f90d-44a9-876d-48245c0fe2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 09:52:51.608495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:51.608465 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" event={"ID":"e9c527fc-edd4-4a04-a781-483921b54c6b","Type":"ContainerStarted","Data":"99845b450fe48ef8f15f68f6928ff521334a9778ee9179df21243739d1e2490f"} Oct 30 09:52:51.608895 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:51.608513 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" event={"ID":"e9c527fc-edd4-4a04-a781-483921b54c6b","Type":"ContainerStarted","Data":"d52085f52da729755d7ac869d91d1c36c26231813b8773d39f73222ef15bed45"} Oct 30 09:52:51.608895 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:51.608653 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" Oct 30 09:52:51.610881 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:51.610854 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" event={"ID":"80bc4ca3-dd13-4fa4-900f-82b93b520ffe","Type":"ContainerStarted","Data":"8fba50c95fdca4091671a1c052d2a56fe007f6f558e474711dd72994e71272e7"} Oct 30 09:52:51.611043 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:51.610925 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" event={"ID":"80bc4ca3-dd13-4fa4-900f-82b93b520ffe","Type":"ContainerStarted","Data":"c1db1804633c6f38bdfa8bc3c417a7a72c768225da6c264728327818556c62a1"} Oct 30 09:52:51.675078 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:51.675008 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" podStartSLOduration=1.674989124 podStartE2EDuration="1.674989124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:52:51.652995676 +0000 UTC m=+3489.515701501" watchObservedRunningTime="2026-10-30 09:52:51.674989124 +0000 UTC m=+3489.537694946" Oct 30 09:52:51.700547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:51.700492 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" podStartSLOduration=1.7004771220000001 podStartE2EDuration="1.700477122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:52:51.696012498 +0000 UTC m=+3489.558718320" watchObservedRunningTime="2026-10-30 09:52:51.700477122 +0000 UTC m=+3489.563182968" Oct 30 09:52:51.906146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:51.906118 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" Oct 30 09:52:52.001472 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:52.001437 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a46a6f-5252-4a98-85be-982321aab480-kube-api-access-58h97\" (UniqueName: \"kubernetes.io/projected/66a46a6f-5252-4a98-85be-982321aab480-kube-api-access-58h97\") pod \"66a46a6f-5252-4a98-85be-982321aab480\" (UID: \"66a46a6f-5252-4a98-85be-982321aab480\") " Oct 30 09:52:52.001671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:52.001611 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66a46a6f-5252-4a98-85be-982321aab480-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a46a6f-5252-4a98-85be-982321aab480-maas-api-openshift-ca\") pod \"66a46a6f-5252-4a98-85be-982321aab480\" (UID: \"66a46a6f-5252-4a98-85be-982321aab480\") " Oct 30 09:52:52.001745 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:52.001719 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66a46a6f-5252-4a98-85be-982321aab480-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a46a6f-5252-4a98-85be-982321aab480-maas-api-xks-ca\") pod \"66a46a6f-5252-4a98-85be-982321aab480\" (UID: \"66a46a6f-5252-4a98-85be-982321aab480\") " Oct 30 09:52:52.002074 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:52.002043 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a46a6f-5252-4a98-85be-982321aab480-maas-api-openshift-ca" pod "66a46a6f-5252-4a98-85be-982321aab480" (UID: "66a46a6f-5252-4a98-85be-982321aab480"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:52:52.002778 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:52.002752 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a46a6f-5252-4a98-85be-982321aab480-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:52:52.004030 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:52.003996 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66a46a6f-5252-4a98-85be-982321aab480-maas-api-xks-ca" pod "66a46a6f-5252-4a98-85be-982321aab480" (UID: "66a46a6f-5252-4a98-85be-982321aab480"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:52:52.004135 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:52.004114 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a46a6f-5252-4a98-85be-982321aab480-kube-api-access-58h97" pod "66a46a6f-5252-4a98-85be-982321aab480" (UID: "66a46a6f-5252-4a98-85be-982321aab480"). InnerVolumeSpecName "kube-api-access-58h97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:52:52.104600 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:52.104567 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58h97\" (UniqueName: \"kubernetes.io/projected/66a46a6f-5252-4a98-85be-982321aab480-kube-api-access-58h97\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:52:52.104769 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:52.104610 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a46a6f-5252-4a98-85be-982321aab480-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:52:52.518354 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:52.518320 5938 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 09:52:52.518530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:52.518369 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" podUID="c7325b5f-f90d-44a9-876d-48245c0fe2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 09:52:52.646193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:52.646160 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" Oct 30 09:52:52.646690 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:52.646201 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds" event={"ID":"66a46a6f-5252-4a98-85be-982321aab480","Type":"ContainerDied","Data":"058255582e3ab02e28b20382d6f3ad85025020c9b944a3dcec05f1b2e9f3b168"} Oct 30 09:52:52.646690 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:52.646263 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="058255582e3ab02e28b20382d6f3ad85025020c9b944a3dcec05f1b2e9f3b168" Oct 30 09:52:52.646690 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:52.646650 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" Oct 30 09:52:53.518128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:53.518098 5938 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 09:52:53.518349 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:53.518148 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" podUID="c7325b5f-f90d-44a9-876d-48245c0fe2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 09:52:54.518281 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:54.518246 5938 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 09:52:54.518803 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:54.518304 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" podUID="c7325b5f-f90d-44a9-876d-48245c0fe2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 09:52:55.518239 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:55.518203 5938 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 09:52:55.518459 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:55.518267 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" podUID="c7325b5f-f90d-44a9-876d-48245c0fe2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 09:52:56.524068 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:56.524036 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:56.524565 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:56.524500 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:56.525298 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:56.525280 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:52:58.029166 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:58.029131 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr"] Oct 30 09:52:58.041848 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:58.041818 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4bb0e57-w2sjr"] Oct 30 09:52:58.259005 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:52:58.258970 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09e48367-fdd0-46b5-b288-be6c743ffa9a" path="/var/lib/kubelet/pods/09e48367-fdd0-46b5-b288-be6c743ffa9a/volumes" Oct 30 09:53:02.649717 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:02.649668 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" Oct 30 09:53:03.657880 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:03.657853 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" Oct 30 09:53:04.532678 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.532650 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw"] Oct 30 09:53:04.533404 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.533388 5938 cpu_manager.go:414] "containerMap: removing container" podUID="66a46a6f-5252-4a98-85be-982321aab480" containerName="revoke-keys" Oct 30 09:53:04.533474 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.533405 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a46a6f-5252-4a98-85be-982321aab480" containerName="revoke-keys" Oct 30 09:53:04.533530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.533520 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a46a6f-5252-4a98-85be-982321aab480" containerName="revoke-keys" Oct 30 09:53:04.538401 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.538378 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" Oct 30 09:53:04.555654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.555629 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw"] Oct 30 09:53:04.564843 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.564817 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:53:04.565085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.565062 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:53:04.643869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.643837 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8736557b-59cd-4e5e-8f14-73ac0cf641c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77d0a328-xbbfw\" (UID: \"8736557b-59cd-4e5e-8f14-73ac0cf641c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" Oct 30 09:53:04.644026 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.643999 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8736557b-59cd-4e5e-8f14-73ac0cf641c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77d0a328-xbbfw\" (UID: \"8736557b-59cd-4e5e-8f14-73ac0cf641c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" Oct 30 09:53:04.644084 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.644067 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmrs9\" (UniqueName: \"kubernetes.io/projected/8736557b-59cd-4e5e-8f14-73ac0cf641c5-kube-api-access-gmrs9\") pod \"maas-api-revoke-sub-77d0a328-xbbfw\" (UID: \"8736557b-59cd-4e5e-8f14-73ac0cf641c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" Oct 30 09:53:04.745103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.745073 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8736557b-59cd-4e5e-8f14-73ac0cf641c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77d0a328-xbbfw\" (UID: \"8736557b-59cd-4e5e-8f14-73ac0cf641c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" Oct 30 09:53:04.745545 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.745271 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8736557b-59cd-4e5e-8f14-73ac0cf641c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77d0a328-xbbfw\" (UID: \"8736557b-59cd-4e5e-8f14-73ac0cf641c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" Oct 30 09:53:04.745545 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.745357 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmrs9\" (UniqueName: \"kubernetes.io/projected/8736557b-59cd-4e5e-8f14-73ac0cf641c5-kube-api-access-gmrs9\") pod \"maas-api-revoke-sub-77d0a328-xbbfw\" (UID: \"8736557b-59cd-4e5e-8f14-73ac0cf641c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" Oct 30 09:53:04.746061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.746029 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8736557b-59cd-4e5e-8f14-73ac0cf641c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77d0a328-xbbfw\" (UID: \"8736557b-59cd-4e5e-8f14-73ac0cf641c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" Oct 30 09:53:04.747663 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.747646 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8736557b-59cd-4e5e-8f14-73ac0cf641c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77d0a328-xbbfw\" (UID: \"8736557b-59cd-4e5e-8f14-73ac0cf641c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" Oct 30 09:53:04.766237 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.766207 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmrs9\" (UniqueName: \"kubernetes.io/projected/8736557b-59cd-4e5e-8f14-73ac0cf641c5-kube-api-access-gmrs9\") pod \"maas-api-revoke-sub-77d0a328-xbbfw\" (UID: \"8736557b-59cd-4e5e-8f14-73ac0cf641c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" Oct 30 09:53:04.856364 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.853509 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" Oct 30 09:53:04.916330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.893832 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz"] Oct 30 09:53:04.916330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.914647 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" Oct 30 09:53:04.931292 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:04.925289 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz"] Oct 30 09:53:05.037267 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.037238 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q"] Oct 30 09:53:05.044377 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.044347 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:05.052296 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.052082 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f9aa00-8f06-49b2-bb82-f749d7960915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c3e7b2b-5mvfz\" (UID: \"c9f9aa00-8f06-49b2-bb82-f749d7960915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" Oct 30 09:53:05.052296 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.052177 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pnfb\" (UniqueName: \"kubernetes.io/projected/c9f9aa00-8f06-49b2-bb82-f749d7960915-kube-api-access-5pnfb\") pod \"maas-api-revoke-sub-0c3e7b2b-5mvfz\" (UID: \"c9f9aa00-8f06-49b2-bb82-f749d7960915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" Oct 30 09:53:05.053146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.052725 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f9aa00-8f06-49b2-bb82-f749d7960915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c3e7b2b-5mvfz\" (UID: \"c9f9aa00-8f06-49b2-bb82-f749d7960915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" Oct 30 09:53:05.060373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.060311 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q"] Oct 30 09:53:05.060671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.060578 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"openshift-service-ca.crt\"" Oct 30 09:53:05.060671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.060635 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"agc-cred-29b8872562de\"" Oct 30 09:53:05.060671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.060581 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"kube-root-ca.crt\"" Oct 30 09:53:05.060955 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.060695 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"payload-processing-external-model-e2e--d06f4d1b-dockercfg-5mx45\"" Oct 30 09:53:05.060955 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.060769 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"routing-overlay\"" Oct 30 09:53:05.060955 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.060855 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"payload-processing-external-model-plugins-e2e-extmod-a-881f83\"" Oct 30 09:53:05.154127 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.154089 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91411246-9ea5-444b-832e-07b3b0344886-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q\" (UID: \"91411246-9ea5-444b-832e-07b3b0344886\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:05.154342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.154180 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f9aa00-8f06-49b2-bb82-f749d7960915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c3e7b2b-5mvfz\" (UID: \"c9f9aa00-8f06-49b2-bb82-f749d7960915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" Oct 30 09:53:05.154342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.154200 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gszfx\" (UniqueName: \"kubernetes.io/projected/91411246-9ea5-444b-832e-07b3b0344886-kube-api-access-gszfx\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q\" (UID: \"91411246-9ea5-444b-832e-07b3b0344886\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:05.154342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.154229 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91411246-9ea5-444b-832e-07b3b0344886-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q\" (UID: \"91411246-9ea5-444b-832e-07b3b0344886\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:05.154342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.154269 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91411246-9ea5-444b-832e-07b3b0344886-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q\" (UID: \"91411246-9ea5-444b-832e-07b3b0344886\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:05.154664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.154401 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f9aa00-8f06-49b2-bb82-f749d7960915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c3e7b2b-5mvfz\" (UID: \"c9f9aa00-8f06-49b2-bb82-f749d7960915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" Oct 30 09:53:05.154664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.154483 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pnfb\" (UniqueName: \"kubernetes.io/projected/c9f9aa00-8f06-49b2-bb82-f749d7960915-kube-api-access-5pnfb\") pod \"maas-api-revoke-sub-0c3e7b2b-5mvfz\" (UID: \"c9f9aa00-8f06-49b2-bb82-f749d7960915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" Oct 30 09:53:05.155767 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.155733 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f9aa00-8f06-49b2-bb82-f749d7960915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c3e7b2b-5mvfz\" (UID: \"c9f9aa00-8f06-49b2-bb82-f749d7960915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" Oct 30 09:53:05.158494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.158469 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f9aa00-8f06-49b2-bb82-f749d7960915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c3e7b2b-5mvfz\" (UID: \"c9f9aa00-8f06-49b2-bb82-f749d7960915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" Oct 30 09:53:05.217360 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.217320 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pnfb\" (UniqueName: \"kubernetes.io/projected/c9f9aa00-8f06-49b2-bb82-f749d7960915-kube-api-access-5pnfb\") pod \"maas-api-revoke-sub-0c3e7b2b-5mvfz\" (UID: \"c9f9aa00-8f06-49b2-bb82-f749d7960915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" Oct 30 09:53:05.255497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.255457 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91411246-9ea5-444b-832e-07b3b0344886-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q\" (UID: \"91411246-9ea5-444b-832e-07b3b0344886\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:05.255825 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.255802 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91411246-9ea5-444b-832e-07b3b0344886-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q\" (UID: \"91411246-9ea5-444b-832e-07b3b0344886\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:05.255931 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.255876 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gszfx\" (UniqueName: \"kubernetes.io/projected/91411246-9ea5-444b-832e-07b3b0344886-kube-api-access-gszfx\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q\" (UID: \"91411246-9ea5-444b-832e-07b3b0344886\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:05.255931 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.255918 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91411246-9ea5-444b-832e-07b3b0344886-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q\" (UID: \"91411246-9ea5-444b-832e-07b3b0344886\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:05.256527 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.256489 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91411246-9ea5-444b-832e-07b3b0344886-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q\" (UID: \"91411246-9ea5-444b-832e-07b3b0344886\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:05.257079 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.257057 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91411246-9ea5-444b-832e-07b3b0344886-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q\" (UID: \"91411246-9ea5-444b-832e-07b3b0344886\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:05.259672 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.259648 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91411246-9ea5-444b-832e-07b3b0344886-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q\" (UID: \"91411246-9ea5-444b-832e-07b3b0344886\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:05.274530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.274490 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw"] Oct 30 09:53:05.277886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.277866 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gszfx\" (UniqueName: \"kubernetes.io/projected/91411246-9ea5-444b-832e-07b3b0344886-kube-api-access-gszfx\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q\" (UID: \"91411246-9ea5-444b-832e-07b3b0344886\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:05.306082 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.306057 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" Oct 30 09:53:05.368842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.368817 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:05.623622 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.623577 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz"] Oct 30 09:53:05.624828 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:53:05.624794 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f9aa00_8f06_49b2_bb82_f749d7960915.slice/crio-c7a96513ae8a858d49c7b401ab0aeef63700bda153bd39399c7abd42f1a71f4d WatchSource:0}: Error finding container c7a96513ae8a858d49c7b401ab0aeef63700bda153bd39399c7abd42f1a71f4d: Status 404 returned error can't find the container with id c7a96513ae8a858d49c7b401ab0aeef63700bda153bd39399c7abd42f1a71f4d Oct 30 09:53:05.690448 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.690422 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q"] Oct 30 09:53:05.691212 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:53:05.691184 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91411246_9ea5_444b_832e_07b3b0344886.slice/crio-1907f18b128be34027aeea671904102e43c54156453a56c7636989dae448c0a8 WatchSource:0}: Error finding container 1907f18b128be34027aeea671904102e43c54156453a56c7636989dae448c0a8: Status 404 returned error can't find the container with id 1907f18b128be34027aeea671904102e43c54156453a56c7636989dae448c0a8 Oct 30 09:53:05.813508 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.813468 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" event={"ID":"91411246-9ea5-444b-832e-07b3b0344886","Type":"ContainerStarted","Data":"1907f18b128be34027aeea671904102e43c54156453a56c7636989dae448c0a8"} Oct 30 09:53:05.815445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.815413 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" event={"ID":"8736557b-59cd-4e5e-8f14-73ac0cf641c5","Type":"ContainerStarted","Data":"916970aa2dc1d8cdff84056a524466344273f94bb89ec33da8adb032073c977d"} Oct 30 09:53:05.815645 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.815467 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" event={"ID":"8736557b-59cd-4e5e-8f14-73ac0cf641c5","Type":"ContainerStarted","Data":"ff7e058ec2125d07403e109b1110df08147836c7ac45598b3aa3eb8afe9b62e8"} Oct 30 09:53:05.816970 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.816940 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" event={"ID":"c9f9aa00-8f06-49b2-bb82-f749d7960915","Type":"ContainerStarted","Data":"c7a96513ae8a858d49c7b401ab0aeef63700bda153bd39399c7abd42f1a71f4d"} Oct 30 09:53:05.877358 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:05.874452 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" podStartSLOduration=1.8744325179999999 podStartE2EDuration="1.874432518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:05.862013766 +0000 UTC m=+3503.724719609" watchObservedRunningTime="2026-10-30 09:53:05.874432518 +0000 UTC m=+3503.737138340" Oct 30 09:53:06.827548 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:06.827513 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" event={"ID":"91411246-9ea5-444b-832e-07b3b0344886","Type":"ContainerStarted","Data":"f4f53e28efbae35d5244ed1c90da4c2016b0ef7fb90b926c04fef332d6f45e34"} Oct 30 09:53:06.828053 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:06.827635 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:06.829775 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:06.829747 5938 generic.go:356] "Generic (PLEG): container finished" podID="8736557b-59cd-4e5e-8f14-73ac0cf641c5" containerID="916970aa2dc1d8cdff84056a524466344273f94bb89ec33da8adb032073c977d" exitCode=0 Oct 30 09:53:06.829932 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:06.829827 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" event={"ID":"8736557b-59cd-4e5e-8f14-73ac0cf641c5","Type":"ContainerDied","Data":"916970aa2dc1d8cdff84056a524466344273f94bb89ec33da8adb032073c977d"} Oct 30 09:53:06.831717 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:06.831692 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" event={"ID":"c9f9aa00-8f06-49b2-bb82-f749d7960915","Type":"ContainerStarted","Data":"e8808b2301dde0ad62942a53e060d911d2890c7f114f85c9f59e48dbd91cea25"} Oct 30 09:53:06.871832 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:06.871790 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" podStartSLOduration=1.8717764639999999 podStartE2EDuration="1.871776464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:06.870028458 +0000 UTC m=+3504.732734269" watchObservedRunningTime="2026-10-30 09:53:06.871776464 +0000 UTC m=+3504.734482286" Oct 30 09:53:07.841277 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:07.841244 5938 generic.go:356] "Generic (PLEG): container finished" podID="c9f9aa00-8f06-49b2-bb82-f749d7960915" containerID="e8808b2301dde0ad62942a53e060d911d2890c7f114f85c9f59e48dbd91cea25" exitCode=0 Oct 30 09:53:07.841785 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:07.841323 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" event={"ID":"c9f9aa00-8f06-49b2-bb82-f749d7960915","Type":"ContainerDied","Data":"e8808b2301dde0ad62942a53e060d911d2890c7f114f85c9f59e48dbd91cea25"} Oct 30 09:53:08.040770 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.040740 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" Oct 30 09:53:08.079188 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.079164 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" Oct 30 09:53:08.103327 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.103262 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8736557b-59cd-4e5e-8f14-73ac0cf641c5-kube-api-access-gmrs9\" (UniqueName: \"kubernetes.io/projected/8736557b-59cd-4e5e-8f14-73ac0cf641c5-kube-api-access-gmrs9\") pod \"8736557b-59cd-4e5e-8f14-73ac0cf641c5\" (UID: \"8736557b-59cd-4e5e-8f14-73ac0cf641c5\") " Oct 30 09:53:08.103465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.103363 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8736557b-59cd-4e5e-8f14-73ac0cf641c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8736557b-59cd-4e5e-8f14-73ac0cf641c5-maas-api-openshift-ca\") pod \"8736557b-59cd-4e5e-8f14-73ac0cf641c5\" (UID: \"8736557b-59cd-4e5e-8f14-73ac0cf641c5\") " Oct 30 09:53:08.103537 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.103489 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8736557b-59cd-4e5e-8f14-73ac0cf641c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8736557b-59cd-4e5e-8f14-73ac0cf641c5-maas-api-xks-ca\") pod \"8736557b-59cd-4e5e-8f14-73ac0cf641c5\" (UID: \"8736557b-59cd-4e5e-8f14-73ac0cf641c5\") " Oct 30 09:53:08.103836 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.103802 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8736557b-59cd-4e5e-8f14-73ac0cf641c5-maas-api-openshift-ca" pod "8736557b-59cd-4e5e-8f14-73ac0cf641c5" (UID: "8736557b-59cd-4e5e-8f14-73ac0cf641c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:08.104452 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.104427 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8736557b-59cd-4e5e-8f14-73ac0cf641c5-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:08.106013 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.105987 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8736557b-59cd-4e5e-8f14-73ac0cf641c5-kube-api-access-gmrs9" pod "8736557b-59cd-4e5e-8f14-73ac0cf641c5" (UID: "8736557b-59cd-4e5e-8f14-73ac0cf641c5"). InnerVolumeSpecName "kube-api-access-gmrs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:08.106324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.106298 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8736557b-59cd-4e5e-8f14-73ac0cf641c5-maas-api-xks-ca" pod "8736557b-59cd-4e5e-8f14-73ac0cf641c5" (UID: "8736557b-59cd-4e5e-8f14-73ac0cf641c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:53:08.206005 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.205969 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f9aa00-8f06-49b2-bb82-f749d7960915-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f9aa00-8f06-49b2-bb82-f749d7960915-maas-api-openshift-ca\") pod \"c9f9aa00-8f06-49b2-bb82-f749d7960915\" (UID: \"c9f9aa00-8f06-49b2-bb82-f749d7960915\") " Oct 30 09:53:08.206164 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.206037 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9f9aa00-8f06-49b2-bb82-f749d7960915-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f9aa00-8f06-49b2-bb82-f749d7960915-maas-api-xks-ca\") pod \"c9f9aa00-8f06-49b2-bb82-f749d7960915\" (UID: \"c9f9aa00-8f06-49b2-bb82-f749d7960915\") " Oct 30 09:53:08.206238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.206223 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f9aa00-8f06-49b2-bb82-f749d7960915-kube-api-access-5pnfb\" (UniqueName: \"kubernetes.io/projected/c9f9aa00-8f06-49b2-bb82-f749d7960915-kube-api-access-5pnfb\") pod \"c9f9aa00-8f06-49b2-bb82-f749d7960915\" (UID: \"c9f9aa00-8f06-49b2-bb82-f749d7960915\") " Oct 30 09:53:08.206338 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.206308 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f9aa00-8f06-49b2-bb82-f749d7960915-maas-api-openshift-ca" pod "c9f9aa00-8f06-49b2-bb82-f749d7960915" (UID: "c9f9aa00-8f06-49b2-bb82-f749d7960915"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:08.206935 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.206871 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8736557b-59cd-4e5e-8f14-73ac0cf641c5-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:08.206935 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.206899 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f9aa00-8f06-49b2-bb82-f749d7960915-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:08.206935 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.206913 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmrs9\" (UniqueName: \"kubernetes.io/projected/8736557b-59cd-4e5e-8f14-73ac0cf641c5-kube-api-access-gmrs9\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:08.208331 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.208306 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9f9aa00-8f06-49b2-bb82-f749d7960915-maas-api-xks-ca" pod "c9f9aa00-8f06-49b2-bb82-f749d7960915" (UID: "c9f9aa00-8f06-49b2-bb82-f749d7960915"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:53:08.208436 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.208418 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f9aa00-8f06-49b2-bb82-f749d7960915-kube-api-access-5pnfb" pod "c9f9aa00-8f06-49b2-bb82-f749d7960915" (UID: "c9f9aa00-8f06-49b2-bb82-f749d7960915"). InnerVolumeSpecName "kube-api-access-5pnfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:08.308326 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.308300 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f9aa00-8f06-49b2-bb82-f749d7960915-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:08.308326 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.308327 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pnfb\" (UniqueName: \"kubernetes.io/projected/c9f9aa00-8f06-49b2-bb82-f749d7960915-kube-api-access-5pnfb\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:08.852091 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.852063 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" Oct 30 09:53:08.852474 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.852099 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz" event={"ID":"c9f9aa00-8f06-49b2-bb82-f749d7960915","Type":"ContainerDied","Data":"c7a96513ae8a858d49c7b401ab0aeef63700bda153bd39399c7abd42f1a71f4d"} Oct 30 09:53:08.852474 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.852148 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7a96513ae8a858d49c7b401ab0aeef63700bda153bd39399c7abd42f1a71f4d" Oct 30 09:53:08.853856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.853830 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" event={"ID":"8736557b-59cd-4e5e-8f14-73ac0cf641c5","Type":"ContainerDied","Data":"ff7e058ec2125d07403e109b1110df08147836c7ac45598b3aa3eb8afe9b62e8"} Oct 30 09:53:08.853856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.853855 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw" Oct 30 09:53:08.854040 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:08.853876 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff7e058ec2125d07403e109b1110df08147836c7ac45598b3aa3eb8afe9b62e8" Oct 30 09:53:09.666643 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:09.666611 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q"] Oct 30 09:53:14.667556 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:14.667503 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" podUID="91411246-9ea5-444b-832e-07b3b0344886" containerName="payload-processing" containerID="cri-o://f4f53e28efbae35d5244ed1c90da4c2016b0ef7fb90b926c04fef332d6f45e34" gracePeriod=25 Oct 30 09:53:14.917000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:14.916966 5938 generic.go:356] "Generic (PLEG): container finished" podID="91411246-9ea5-444b-832e-07b3b0344886" containerID="f4f53e28efbae35d5244ed1c90da4c2016b0ef7fb90b926c04fef332d6f45e34" exitCode=0 Oct 30 09:53:14.917220 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:14.917201 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" event={"ID":"91411246-9ea5-444b-832e-07b3b0344886","Type":"ContainerDied","Data":"f4f53e28efbae35d5244ed1c90da4c2016b0ef7fb90b926c04fef332d6f45e34"} Oct 30 09:53:14.943221 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:14.943198 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:15.009792 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.009758 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91411246-9ea5-444b-832e-07b3b0344886-provider-credentials\" (UniqueName: \"kubernetes.io/projected/91411246-9ea5-444b-832e-07b3b0344886-provider-credentials\") pod \"91411246-9ea5-444b-832e-07b3b0344886\" (UID: \"91411246-9ea5-444b-832e-07b3b0344886\") " Oct 30 09:53:15.009949 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.009811 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91411246-9ea5-444b-832e-07b3b0344886-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91411246-9ea5-444b-832e-07b3b0344886-plugins-config-volume\") pod \"91411246-9ea5-444b-832e-07b3b0344886\" (UID: \"91411246-9ea5-444b-832e-07b3b0344886\") " Oct 30 09:53:15.009949 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.009869 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91411246-9ea5-444b-832e-07b3b0344886-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91411246-9ea5-444b-832e-07b3b0344886-routing-overlay-volume\") pod \"91411246-9ea5-444b-832e-07b3b0344886\" (UID: \"91411246-9ea5-444b-832e-07b3b0344886\") " Oct 30 09:53:15.010277 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.010162 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91411246-9ea5-444b-832e-07b3b0344886-kube-api-access-gszfx\" (UniqueName: \"kubernetes.io/projected/91411246-9ea5-444b-832e-07b3b0344886-kube-api-access-gszfx\") pod \"91411246-9ea5-444b-832e-07b3b0344886\" (UID: \"91411246-9ea5-444b-832e-07b3b0344886\") " Oct 30 09:53:15.010277 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.010223 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91411246-9ea5-444b-832e-07b3b0344886-plugins-config-volume" pod "91411246-9ea5-444b-832e-07b3b0344886" (UID: "91411246-9ea5-444b-832e-07b3b0344886"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:15.010277 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.010251 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91411246-9ea5-444b-832e-07b3b0344886-routing-overlay-volume" pod "91411246-9ea5-444b-832e-07b3b0344886" (UID: "91411246-9ea5-444b-832e-07b3b0344886"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:15.011144 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.011117 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91411246-9ea5-444b-832e-07b3b0344886-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:15.011266 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.011150 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91411246-9ea5-444b-832e-07b3b0344886-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:15.012440 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.012417 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91411246-9ea5-444b-832e-07b3b0344886-provider-credentials" pod "91411246-9ea5-444b-832e-07b3b0344886" (UID: "91411246-9ea5-444b-832e-07b3b0344886"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:15.012532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.012463 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91411246-9ea5-444b-832e-07b3b0344886-kube-api-access-gszfx" pod "91411246-9ea5-444b-832e-07b3b0344886" (UID: "91411246-9ea5-444b-832e-07b3b0344886"). InnerVolumeSpecName "kube-api-access-gszfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:15.112943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.112908 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gszfx\" (UniqueName: \"kubernetes.io/projected/91411246-9ea5-444b-832e-07b3b0344886-kube-api-access-gszfx\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:15.112943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.112935 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91411246-9ea5-444b-832e-07b3b0344886-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:15.114892 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.114872 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b"] Oct 30 09:53:15.115638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.115622 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8736557b-59cd-4e5e-8f14-73ac0cf641c5" containerName="revoke-keys" Oct 30 09:53:15.115689 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.115641 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8736557b-59cd-4e5e-8f14-73ac0cf641c5" containerName="revoke-keys" Oct 30 09:53:15.115689 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.115659 5938 cpu_manager.go:414] "containerMap: removing container" podUID="91411246-9ea5-444b-832e-07b3b0344886" containerName="payload-processing" Oct 30 09:53:15.115689 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.115664 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91411246-9ea5-444b-832e-07b3b0344886" containerName="payload-processing" Oct 30 09:53:15.115798 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.115698 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c9f9aa00-8f06-49b2-bb82-f749d7960915" containerName="revoke-keys" Oct 30 09:53:15.115798 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.115703 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f9aa00-8f06-49b2-bb82-f749d7960915" containerName="revoke-keys" Oct 30 09:53:15.115857 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.115845 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f9aa00-8f06-49b2-bb82-f749d7960915" containerName="revoke-keys" Oct 30 09:53:15.115895 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.115859 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8736557b-59cd-4e5e-8f14-73ac0cf641c5" containerName="revoke-keys" Oct 30 09:53:15.115895 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.115865 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="91411246-9ea5-444b-832e-07b3b0344886" containerName="payload-processing" Oct 30 09:53:15.119693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.119679 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b" Oct 30 09:53:15.136521 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.136489 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:53:15.136521 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.136493 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:53:15.139986 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.139966 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b"] Oct 30 09:53:15.214183 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.214109 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/375796be-d502-4249-bf0a-9e32ab37aefa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39b8bd03-dnq2b\" (UID: \"375796be-d502-4249-bf0a-9e32ab37aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b" Oct 30 09:53:15.214183 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.214159 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/375796be-d502-4249-bf0a-9e32ab37aefa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39b8bd03-dnq2b\" (UID: \"375796be-d502-4249-bf0a-9e32ab37aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b" Oct 30 09:53:15.214374 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.214284 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nnlm\" (UniqueName: \"kubernetes.io/projected/375796be-d502-4249-bf0a-9e32ab37aefa-kube-api-access-2nnlm\") pod \"maas-api-revoke-sub-39b8bd03-dnq2b\" (UID: \"375796be-d502-4249-bf0a-9e32ab37aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b" Oct 30 09:53:15.315924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.315898 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nnlm\" (UniqueName: \"kubernetes.io/projected/375796be-d502-4249-bf0a-9e32ab37aefa-kube-api-access-2nnlm\") pod \"maas-api-revoke-sub-39b8bd03-dnq2b\" (UID: \"375796be-d502-4249-bf0a-9e32ab37aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b" Oct 30 09:53:15.316246 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.316222 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/375796be-d502-4249-bf0a-9e32ab37aefa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39b8bd03-dnq2b\" (UID: \"375796be-d502-4249-bf0a-9e32ab37aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b" Oct 30 09:53:15.316368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.316288 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/375796be-d502-4249-bf0a-9e32ab37aefa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39b8bd03-dnq2b\" (UID: \"375796be-d502-4249-bf0a-9e32ab37aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b" Oct 30 09:53:15.317038 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.317015 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/375796be-d502-4249-bf0a-9e32ab37aefa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39b8bd03-dnq2b\" (UID: \"375796be-d502-4249-bf0a-9e32ab37aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b" Oct 30 09:53:15.318661 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.318643 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/375796be-d502-4249-bf0a-9e32ab37aefa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39b8bd03-dnq2b\" (UID: \"375796be-d502-4249-bf0a-9e32ab37aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b" Oct 30 09:53:15.336130 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.336098 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nnlm\" (UniqueName: \"kubernetes.io/projected/375796be-d502-4249-bf0a-9e32ab37aefa-kube-api-access-2nnlm\") pod \"maas-api-revoke-sub-39b8bd03-dnq2b\" (UID: \"375796be-d502-4249-bf0a-9e32ab37aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b" Oct 30 09:53:15.434264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.434229 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b" Oct 30 09:53:15.695952 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.695923 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw"] Oct 30 09:53:15.700813 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.700791 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" Oct 30 09:53:15.717410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.717380 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw"] Oct 30 09:53:15.722789 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.722764 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb81527-7ae4-4642-91b8-26160d49ac55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b6e78aa-phqvw\" (UID: \"5cb81527-7ae4-4642-91b8-26160d49ac55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" Oct 30 09:53:15.722947 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.722924 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6smg\" (UniqueName: \"kubernetes.io/projected/5cb81527-7ae4-4642-91b8-26160d49ac55-kube-api-access-m6smg\") pod \"maas-api-revoke-sub-3b6e78aa-phqvw\" (UID: \"5cb81527-7ae4-4642-91b8-26160d49ac55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" Oct 30 09:53:15.723084 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.723067 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb81527-7ae4-4642-91b8-26160d49ac55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b6e78aa-phqvw\" (UID: \"5cb81527-7ae4-4642-91b8-26160d49ac55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" Oct 30 09:53:15.740422 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.740392 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b"] Oct 30 09:53:15.745427 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:53:15.745400 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375796be_d502_4249_bf0a_9e32ab37aefa.slice/crio-cbf25a8c78b6093b3540e0828aff01ee6cc4097c5da64e43e5a6cda823f41df9 WatchSource:0}: Error finding container cbf25a8c78b6093b3540e0828aff01ee6cc4097c5da64e43e5a6cda823f41df9: Status 404 returned error can't find the container with id cbf25a8c78b6093b3540e0828aff01ee6cc4097c5da64e43e5a6cda823f41df9 Oct 30 09:53:15.812125 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:53:15.812092 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:53:15.824567 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.824539 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb81527-7ae4-4642-91b8-26160d49ac55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b6e78aa-phqvw\" (UID: \"5cb81527-7ae4-4642-91b8-26160d49ac55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" Oct 30 09:53:15.824728 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.824711 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb81527-7ae4-4642-91b8-26160d49ac55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b6e78aa-phqvw\" (UID: \"5cb81527-7ae4-4642-91b8-26160d49ac55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" Oct 30 09:53:15.824813 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.824801 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6smg\" (UniqueName: \"kubernetes.io/projected/5cb81527-7ae4-4642-91b8-26160d49ac55-kube-api-access-m6smg\") pod \"maas-api-revoke-sub-3b6e78aa-phqvw\" (UID: \"5cb81527-7ae4-4642-91b8-26160d49ac55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" Oct 30 09:53:15.825249 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.825220 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb81527-7ae4-4642-91b8-26160d49ac55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b6e78aa-phqvw\" (UID: \"5cb81527-7ae4-4642-91b8-26160d49ac55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" Oct 30 09:53:15.827043 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.827023 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb81527-7ae4-4642-91b8-26160d49ac55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b6e78aa-phqvw\" (UID: \"5cb81527-7ae4-4642-91b8-26160d49ac55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" Oct 30 09:53:15.845167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.845145 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6smg\" (UniqueName: \"kubernetes.io/projected/5cb81527-7ae4-4642-91b8-26160d49ac55-kube-api-access-m6smg\") pod \"maas-api-revoke-sub-3b6e78aa-phqvw\" (UID: \"5cb81527-7ae4-4642-91b8-26160d49ac55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" Oct 30 09:53:15.926945 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.926925 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" Oct 30 09:53:15.926945 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.926931 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q" event={"ID":"91411246-9ea5-444b-832e-07b3b0344886","Type":"ContainerDied","Data":"1907f18b128be34027aeea671904102e43c54156453a56c7636989dae448c0a8"} Oct 30 09:53:15.927125 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.927025 5938 scope.go:122] "RemoveContainer" containerID="f4f53e28efbae35d5244ed1c90da4c2016b0ef7fb90b926c04fef332d6f45e34" Oct 30 09:53:15.928477 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.928444 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b" event={"ID":"375796be-d502-4249-bf0a-9e32ab37aefa","Type":"ContainerStarted","Data":"cbf25a8c78b6093b3540e0828aff01ee6cc4097c5da64e43e5a6cda823f41df9"} Oct 30 09:53:15.986076 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.986046 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q"] Oct 30 09:53:15.989325 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:15.989303 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-85d5ln28q"] Oct 30 09:53:16.016082 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.016026 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" Oct 30 09:53:16.260747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.260716 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91411246-9ea5-444b-832e-07b3b0344886" path="/var/lib/kubelet/pods/91411246-9ea5-444b-832e-07b3b0344886/volumes" Oct 30 09:53:16.388279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.388252 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw"] Oct 30 09:53:16.389115 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:53:16.389093 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb81527_7ae4_4642_91b8_26160d49ac55.slice/crio-0a7aadf76e4f31c4f909f8fa86bff6a2138a6b6fe2e0751ce3705ee3a2c6084b WatchSource:0}: Error finding container 0a7aadf76e4f31c4f909f8fa86bff6a2138a6b6fe2e0751ce3705ee3a2c6084b: Status 404 returned error can't find the container with id 0a7aadf76e4f31c4f909f8fa86bff6a2138a6b6fe2e0751ce3705ee3a2c6084b Oct 30 09:53:16.751527 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.751487 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f"] Oct 30 09:53:16.756318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.756293 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" Oct 30 09:53:16.773334 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.773310 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f"] Oct 30 09:53:16.859057 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.859029 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ee945-e9e0-4d97-84a2-b31da7fe494b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f\" (UID: \"f58ee945-e9e0-4d97-84a2-b31da7fe494b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" Oct 30 09:53:16.859240 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.859221 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b4g5\" (UniqueName: \"kubernetes.io/projected/f58ee945-e9e0-4d97-84a2-b31da7fe494b-kube-api-access-5b4g5\") pod \"maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f\" (UID: \"f58ee945-e9e0-4d97-84a2-b31da7fe494b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" Oct 30 09:53:16.859304 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.859289 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ee945-e9e0-4d97-84a2-b31da7fe494b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f\" (UID: \"f58ee945-e9e0-4d97-84a2-b31da7fe494b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" Oct 30 09:53:16.940189 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.940120 5938 generic.go:356] "Generic (PLEG): container finished" podID="375796be-d502-4249-bf0a-9e32ab37aefa" containerID="a4a4413ca29138c58f17837ca209c9a7cef9692db604445d8384a7d534d4f25d" exitCode=0 Oct 30 09:53:16.940311 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.940200 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b" event={"ID":"375796be-d502-4249-bf0a-9e32ab37aefa","Type":"ContainerDied","Data":"a4a4413ca29138c58f17837ca209c9a7cef9692db604445d8384a7d534d4f25d"} Oct 30 09:53:16.942029 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.941994 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" event={"ID":"5cb81527-7ae4-4642-91b8-26160d49ac55","Type":"ContainerStarted","Data":"4a59bdd55838946351d7f21d20c13a928c1f6b3f91ef18750e4a5812de49365c"} Oct 30 09:53:16.942173 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.942050 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" event={"ID":"5cb81527-7ae4-4642-91b8-26160d49ac55","Type":"ContainerStarted","Data":"0a7aadf76e4f31c4f909f8fa86bff6a2138a6b6fe2e0751ce3705ee3a2c6084b"} Oct 30 09:53:16.960523 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.960495 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b4g5\" (UniqueName: \"kubernetes.io/projected/f58ee945-e9e0-4d97-84a2-b31da7fe494b-kube-api-access-5b4g5\") pod \"maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f\" (UID: \"f58ee945-e9e0-4d97-84a2-b31da7fe494b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" Oct 30 09:53:16.960680 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.960571 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ee945-e9e0-4d97-84a2-b31da7fe494b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f\" (UID: \"f58ee945-e9e0-4d97-84a2-b31da7fe494b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" Oct 30 09:53:16.960796 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.960780 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ee945-e9e0-4d97-84a2-b31da7fe494b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f\" (UID: \"f58ee945-e9e0-4d97-84a2-b31da7fe494b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" Oct 30 09:53:16.961307 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.961286 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ee945-e9e0-4d97-84a2-b31da7fe494b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f\" (UID: \"f58ee945-e9e0-4d97-84a2-b31da7fe494b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" Oct 30 09:53:16.963157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.963134 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ee945-e9e0-4d97-84a2-b31da7fe494b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f\" (UID: \"f58ee945-e9e0-4d97-84a2-b31da7fe494b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" Oct 30 09:53:16.981018 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:16.980998 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b4g5\" (UniqueName: \"kubernetes.io/projected/f58ee945-e9e0-4d97-84a2-b31da7fe494b-kube-api-access-5b4g5\") pod \"maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f\" (UID: \"f58ee945-e9e0-4d97-84a2-b31da7fe494b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" Oct 30 09:53:17.024861 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:17.024822 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" podStartSLOduration=2.024809428 podStartE2EDuration="2.024809428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:17.023323915 +0000 UTC m=+3514.886029736" watchObservedRunningTime="2026-10-30 09:53:17.024809428 +0000 UTC m=+3514.887515300" Oct 30 09:53:17.127413 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:17.127386 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" Oct 30 09:53:17.431107 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:17.431081 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f"] Oct 30 09:53:17.431522 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:53:17.431490 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58ee945_e9e0_4d97_84a2_b31da7fe494b.slice/crio-cec5b4bded07aadd37d2c1b94f88706818c888445388a3570166184320f5408d WatchSource:0}: Error finding container cec5b4bded07aadd37d2c1b94f88706818c888445388a3570166184320f5408d: Status 404 returned error can't find the container with id cec5b4bded07aadd37d2c1b94f88706818c888445388a3570166184320f5408d Oct 30 09:53:17.953240 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:17.953205 5938 generic.go:356] "Generic (PLEG): container finished" podID="5cb81527-7ae4-4642-91b8-26160d49ac55" containerID="4a59bdd55838946351d7f21d20c13a928c1f6b3f91ef18750e4a5812de49365c" exitCode=0 Oct 30 09:53:17.953702 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:17.953295 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" event={"ID":"5cb81527-7ae4-4642-91b8-26160d49ac55","Type":"ContainerDied","Data":"4a59bdd55838946351d7f21d20c13a928c1f6b3f91ef18750e4a5812de49365c"} Oct 30 09:53:17.955254 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:17.955227 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" event={"ID":"f58ee945-e9e0-4d97-84a2-b31da7fe494b","Type":"ContainerStarted","Data":"80392f2d2b3b9eb511a6918de3cab20f06e4eee48fae40e707963f3c361b5dd2"} Oct 30 09:53:17.955383 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:17.955267 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" event={"ID":"f58ee945-e9e0-4d97-84a2-b31da7fe494b","Type":"ContainerStarted","Data":"cec5b4bded07aadd37d2c1b94f88706818c888445388a3570166184320f5408d"} Oct 30 09:53:18.034913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:18.034849 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" podStartSLOduration=2.034828668 podStartE2EDuration="2.034828668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:18.033461706 +0000 UTC m=+3515.896167533" watchObservedRunningTime="2026-10-30 09:53:18.034828668 +0000 UTC m=+3515.897534491" Oct 30 09:53:18.136181 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:18.136155 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b" Oct 30 09:53:18.180780 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:18.180752 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375796be-d502-4249-bf0a-9e32ab37aefa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/375796be-d502-4249-bf0a-9e32ab37aefa-maas-api-xks-ca\") pod \"375796be-d502-4249-bf0a-9e32ab37aefa\" (UID: \"375796be-d502-4249-bf0a-9e32ab37aefa\") " Oct 30 09:53:18.180996 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:18.180978 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375796be-d502-4249-bf0a-9e32ab37aefa-kube-api-access-2nnlm\" (UniqueName: \"kubernetes.io/projected/375796be-d502-4249-bf0a-9e32ab37aefa-kube-api-access-2nnlm\") pod \"375796be-d502-4249-bf0a-9e32ab37aefa\" (UID: \"375796be-d502-4249-bf0a-9e32ab37aefa\") " Oct 30 09:53:18.181083 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:18.181067 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375796be-d502-4249-bf0a-9e32ab37aefa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/375796be-d502-4249-bf0a-9e32ab37aefa-maas-api-openshift-ca\") pod \"375796be-d502-4249-bf0a-9e32ab37aefa\" (UID: \"375796be-d502-4249-bf0a-9e32ab37aefa\") " Oct 30 09:53:18.181483 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:18.181404 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375796be-d502-4249-bf0a-9e32ab37aefa-maas-api-openshift-ca" pod "375796be-d502-4249-bf0a-9e32ab37aefa" (UID: "375796be-d502-4249-bf0a-9e32ab37aefa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:18.181704 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:18.181684 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/375796be-d502-4249-bf0a-9e32ab37aefa-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:18.182933 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:18.182913 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375796be-d502-4249-bf0a-9e32ab37aefa-maas-api-xks-ca" pod "375796be-d502-4249-bf0a-9e32ab37aefa" (UID: "375796be-d502-4249-bf0a-9e32ab37aefa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:53:18.183062 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:18.183046 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375796be-d502-4249-bf0a-9e32ab37aefa-kube-api-access-2nnlm" pod "375796be-d502-4249-bf0a-9e32ab37aefa" (UID: "375796be-d502-4249-bf0a-9e32ab37aefa"). InnerVolumeSpecName "kube-api-access-2nnlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:18.282698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:18.282637 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/375796be-d502-4249-bf0a-9e32ab37aefa-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:18.282698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:18.282659 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nnlm\" (UniqueName: \"kubernetes.io/projected/375796be-d502-4249-bf0a-9e32ab37aefa-kube-api-access-2nnlm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:18.964773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:18.964689 5938 generic.go:356] "Generic (PLEG): container finished" podID="f58ee945-e9e0-4d97-84a2-b31da7fe494b" containerID="80392f2d2b3b9eb511a6918de3cab20f06e4eee48fae40e707963f3c361b5dd2" exitCode=0 Oct 30 09:53:18.965142 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:18.964774 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" event={"ID":"f58ee945-e9e0-4d97-84a2-b31da7fe494b","Type":"ContainerDied","Data":"80392f2d2b3b9eb511a6918de3cab20f06e4eee48fae40e707963f3c361b5dd2"} Oct 30 09:53:18.966346 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:18.966331 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b" Oct 30 09:53:18.966392 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:18.966370 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b" event={"ID":"375796be-d502-4249-bf0a-9e32ab37aefa","Type":"ContainerDied","Data":"cbf25a8c78b6093b3540e0828aff01ee6cc4097c5da64e43e5a6cda823f41df9"} Oct 30 09:53:18.966437 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:18.966407 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbf25a8c78b6093b3540e0828aff01ee6cc4097c5da64e43e5a6cda823f41df9" Oct 30 09:53:19.141738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:19.141705 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" Oct 30 09:53:19.194268 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:19.194236 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cb81527-7ae4-4642-91b8-26160d49ac55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb81527-7ae4-4642-91b8-26160d49ac55-maas-api-openshift-ca\") pod \"5cb81527-7ae4-4642-91b8-26160d49ac55\" (UID: \"5cb81527-7ae4-4642-91b8-26160d49ac55\") " Oct 30 09:53:19.194456 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:19.194364 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb81527-7ae4-4642-91b8-26160d49ac55-kube-api-access-m6smg\" (UniqueName: \"kubernetes.io/projected/5cb81527-7ae4-4642-91b8-26160d49ac55-kube-api-access-m6smg\") pod \"5cb81527-7ae4-4642-91b8-26160d49ac55\" (UID: \"5cb81527-7ae4-4642-91b8-26160d49ac55\") " Oct 30 09:53:19.194544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:19.194526 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cb81527-7ae4-4642-91b8-26160d49ac55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb81527-7ae4-4642-91b8-26160d49ac55-maas-api-xks-ca\") pod \"5cb81527-7ae4-4642-91b8-26160d49ac55\" (UID: \"5cb81527-7ae4-4642-91b8-26160d49ac55\") " Oct 30 09:53:19.194666 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:19.194640 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb81527-7ae4-4642-91b8-26160d49ac55-maas-api-openshift-ca" pod "5cb81527-7ae4-4642-91b8-26160d49ac55" (UID: "5cb81527-7ae4-4642-91b8-26160d49ac55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:19.195346 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:19.195320 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb81527-7ae4-4642-91b8-26160d49ac55-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:19.197568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:19.197542 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cb81527-7ae4-4642-91b8-26160d49ac55-maas-api-xks-ca" pod "5cb81527-7ae4-4642-91b8-26160d49ac55" (UID: "5cb81527-7ae4-4642-91b8-26160d49ac55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:53:19.197794 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:19.197614 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb81527-7ae4-4642-91b8-26160d49ac55-kube-api-access-m6smg" pod "5cb81527-7ae4-4642-91b8-26160d49ac55" (UID: "5cb81527-7ae4-4642-91b8-26160d49ac55"). InnerVolumeSpecName "kube-api-access-m6smg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:19.302559 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:19.302468 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb81527-7ae4-4642-91b8-26160d49ac55-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:19.302559 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:19.302500 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6smg\" (UniqueName: \"kubernetes.io/projected/5cb81527-7ae4-4642-91b8-26160d49ac55-kube-api-access-m6smg\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:19.975120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:19.975086 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" Oct 30 09:53:19.975515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:19.975125 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw" event={"ID":"5cb81527-7ae4-4642-91b8-26160d49ac55","Type":"ContainerDied","Data":"0a7aadf76e4f31c4f909f8fa86bff6a2138a6b6fe2e0751ce3705ee3a2c6084b"} Oct 30 09:53:19.975515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:19.975198 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a7aadf76e4f31c4f909f8fa86bff6a2138a6b6fe2e0751ce3705ee3a2c6084b" Oct 30 09:53:20.137895 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:20.137868 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" Oct 30 09:53:20.215840 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:20.215814 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58ee945-e9e0-4d97-84a2-b31da7fe494b-kube-api-access-5b4g5\" (UniqueName: \"kubernetes.io/projected/f58ee945-e9e0-4d97-84a2-b31da7fe494b-kube-api-access-5b4g5\") pod \"f58ee945-e9e0-4d97-84a2-b31da7fe494b\" (UID: \"f58ee945-e9e0-4d97-84a2-b31da7fe494b\") " Oct 30 09:53:20.216050 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:20.216033 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58ee945-e9e0-4d97-84a2-b31da7fe494b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ee945-e9e0-4d97-84a2-b31da7fe494b-maas-api-openshift-ca\") pod \"f58ee945-e9e0-4d97-84a2-b31da7fe494b\" (UID: \"f58ee945-e9e0-4d97-84a2-b31da7fe494b\") " Oct 30 09:53:20.216125 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:20.216113 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58ee945-e9e0-4d97-84a2-b31da7fe494b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ee945-e9e0-4d97-84a2-b31da7fe494b-maas-api-xks-ca\") pod \"f58ee945-e9e0-4d97-84a2-b31da7fe494b\" (UID: \"f58ee945-e9e0-4d97-84a2-b31da7fe494b\") " Oct 30 09:53:20.216472 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:20.216440 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58ee945-e9e0-4d97-84a2-b31da7fe494b-maas-api-openshift-ca" pod "f58ee945-e9e0-4d97-84a2-b31da7fe494b" (UID: "f58ee945-e9e0-4d97-84a2-b31da7fe494b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:20.216833 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:20.216809 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ee945-e9e0-4d97-84a2-b31da7fe494b-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:20.218236 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:20.218212 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58ee945-e9e0-4d97-84a2-b31da7fe494b-kube-api-access-5b4g5" pod "f58ee945-e9e0-4d97-84a2-b31da7fe494b" (UID: "f58ee945-e9e0-4d97-84a2-b31da7fe494b"). InnerVolumeSpecName "kube-api-access-5b4g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:20.218851 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:20.218831 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58ee945-e9e0-4d97-84a2-b31da7fe494b-maas-api-xks-ca" pod "f58ee945-e9e0-4d97-84a2-b31da7fe494b" (UID: "f58ee945-e9e0-4d97-84a2-b31da7fe494b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:53:20.318690 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:20.318666 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ee945-e9e0-4d97-84a2-b31da7fe494b-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:20.318690 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:20.318688 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b4g5\" (UniqueName: \"kubernetes.io/projected/f58ee945-e9e0-4d97-84a2-b31da7fe494b-kube-api-access-5b4g5\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:20.983909 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:20.983885 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" Oct 30 09:53:20.984305 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:20.983927 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f" event={"ID":"f58ee945-e9e0-4d97-84a2-b31da7fe494b","Type":"ContainerDied","Data":"cec5b4bded07aadd37d2c1b94f88706818c888445388a3570166184320f5408d"} Oct 30 09:53:20.984305 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:20.983978 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cec5b4bded07aadd37d2c1b94f88706818c888445388a3570166184320f5408d" Oct 30 09:53:22.757394 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.757314 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw"] Oct 30 09:53:22.758109 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.758091 5938 cpu_manager.go:414] "containerMap: removing container" podUID="375796be-d502-4249-bf0a-9e32ab37aefa" containerName="revoke-keys" Oct 30 09:53:22.758157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.758111 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375796be-d502-4249-bf0a-9e32ab37aefa" containerName="revoke-keys" Oct 30 09:53:22.758157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.758150 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f58ee945-e9e0-4d97-84a2-b31da7fe494b" containerName="revoke-keys" Oct 30 09:53:22.758157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.758156 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58ee945-e9e0-4d97-84a2-b31da7fe494b" containerName="revoke-keys" Oct 30 09:53:22.758271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.758178 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5cb81527-7ae4-4642-91b8-26160d49ac55" containerName="revoke-keys" Oct 30 09:53:22.758271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.758183 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb81527-7ae4-4642-91b8-26160d49ac55" containerName="revoke-keys" Oct 30 09:53:22.758349 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.758338 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58ee945-e9e0-4d97-84a2-b31da7fe494b" containerName="revoke-keys" Oct 30 09:53:22.758385 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.758354 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb81527-7ae4-4642-91b8-26160d49ac55" containerName="revoke-keys" Oct 30 09:53:22.758385 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.758366 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="375796be-d502-4249-bf0a-9e32ab37aefa" containerName="revoke-keys" Oct 30 09:53:22.762278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.762263 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.828764 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.828013 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d66d22-openshift-default-dockercfg-zf9tn\"" Oct 30 09:53:22.834524 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.834492 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw"] Oct 30 09:53:22.850468 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.850436 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-data\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.850639 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.850487 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-envoy\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.850700 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.850644 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-token\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.850700 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.850686 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-workload-certs\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.850842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.850822 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxdn4\" (UniqueName: \"kubernetes.io/projected/df118d0b-7613-4edf-aaa4-af1944531fa6-kube-api-access-hxdn4\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.851064 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.851041 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-workload-socket\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.851174 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.851155 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df118d0b-7613-4edf-aaa4-af1944531fa6-istiod-ca-cert\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.851267 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.851251 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-podinfo\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.851356 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.851335 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-credential-socket\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.952747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.952716 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-data\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.952916 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.952761 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-envoy\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.952916 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.952812 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-token\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.952916 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.952840 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-workload-certs\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.953091 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.952914 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxdn4\" (UniqueName: \"kubernetes.io/projected/df118d0b-7613-4edf-aaa4-af1944531fa6-kube-api-access-hxdn4\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.953091 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.953055 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-workload-socket\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.953091 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.953065 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-data\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.953265 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.953156 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df118d0b-7613-4edf-aaa4-af1944531fa6-istiod-ca-cert\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.953265 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.953230 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-workload-certs\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.953265 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.953240 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-podinfo\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.953445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.953279 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-credential-socket\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.953500 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.953439 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-workload-socket\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.953647 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.953627 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-credential-socket\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.954120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.954085 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df118d0b-7613-4edf-aaa4-af1944531fa6-istiod-ca-cert\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.955404 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.955382 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-envoy\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.955729 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.955711 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-podinfo\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.972738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.972714 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-token\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:22.972835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:22.972795 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxdn4\" (UniqueName: \"kubernetes.io/projected/df118d0b-7613-4edf-aaa4-af1944531fa6-kube-api-access-hxdn4\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:23.076820 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:23.076757 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:23.400386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:23.400357 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw"] Oct 30 09:53:23.401365 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:53:23.401332 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf118d0b_7613_4edf_aaa4_af1944531fa6.slice/crio-5c66bb4b56158cd292228b1d3c96bbf1c2039b1c823e68fdf7fe33cb7196b780 WatchSource:0}: Error finding container 5c66bb4b56158cd292228b1d3c96bbf1c2039b1c823e68fdf7fe33cb7196b780: Status 404 returned error can't find the container with id 5c66bb4b56158cd292228b1d3c96bbf1c2039b1c823e68fdf7fe33cb7196b780 Oct 30 09:53:23.405255 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:23.405222 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:53:23.405356 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:23.405285 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:53:23.405356 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:23.405317 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 09:53:24.011708 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:24.011664 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" event={"ID":"df118d0b-7613-4edf-aaa4-af1944531fa6","Type":"ContainerStarted","Data":"e6304487555bea67797bf74ea39ed434367bb1dfb0daa7232cb858002603102e"} Oct 30 09:53:24.012341 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:24.012292 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" event={"ID":"df118d0b-7613-4edf-aaa4-af1944531fa6","Type":"ContainerStarted","Data":"5c66bb4b56158cd292228b1d3c96bbf1c2039b1c823e68fdf7fe33cb7196b780"} Oct 30 09:53:24.077174 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:24.077136 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:24.078941 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:24.078913 5938 patch_prober.go:29] interesting pod/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 09:53:24.079081 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:24.078970 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" podUID="df118d0b-7613-4edf-aaa4-af1944531fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 09:53:24.082299 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:24.082260 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" podStartSLOduration=2.082247567 podStartE2EDuration="2.082247567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:24.068750186 +0000 UTC m=+3521.931456007" watchObservedRunningTime="2026-10-30 09:53:24.082247567 +0000 UTC m=+3521.944953388" Oct 30 09:53:25.077783 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:25.077755 5938 patch_prober.go:29] interesting pod/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 09:53:25.078191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:25.077800 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" podUID="df118d0b-7613-4edf-aaa4-af1944531fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 09:53:26.077942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:26.077904 5938 patch_prober.go:29] interesting pod/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 09:53:26.078467 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:26.077962 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" podUID="df118d0b-7613-4edf-aaa4-af1944531fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 09:53:27.077875 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.077842 5938 patch_prober.go:29] interesting pod/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 09:53:27.078060 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.077894 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" podUID="df118d0b-7613-4edf-aaa4-af1944531fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 09:53:27.084575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.084549 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp"] Oct 30 09:53:27.090091 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.090072 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:27.107667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.107643 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"agc-cred-7655d9e07e16\"" Oct 30 09:53:27.111879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.111850 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp"] Oct 30 09:53:27.126983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.126959 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"payload-processing-external-model-e2e--2110ea34-dockercfg-68rjw\"" Oct 30 09:53:27.149689 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.142863 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"kube-root-ca.crt\"" Oct 30 09:53:27.149852 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.142969 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d60c68\"" Oct 30 09:53:27.149931 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.143032 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"routing-overlay\"" Oct 30 09:53:27.150009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.143733 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"openshift-service-ca.crt\"" Oct 30 09:53:27.247656 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.247607 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7900e2-f696-43e4-a0c9-c72bac59be25-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp\" (UID: \"4b7900e2-f696-43e4-a0c9-c72bac59be25\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:27.247854 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.247773 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b7900e2-f696-43e4-a0c9-c72bac59be25-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp\" (UID: \"4b7900e2-f696-43e4-a0c9-c72bac59be25\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:27.247922 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.247891 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwqx5\" (UniqueName: \"kubernetes.io/projected/4b7900e2-f696-43e4-a0c9-c72bac59be25-kube-api-access-mwqx5\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp\" (UID: \"4b7900e2-f696-43e4-a0c9-c72bac59be25\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:27.248135 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.248113 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b7900e2-f696-43e4-a0c9-c72bac59be25-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp\" (UID: \"4b7900e2-f696-43e4-a0c9-c72bac59be25\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:27.344827 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.344722 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7"] Oct 30 09:53:27.353378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.349875 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b7900e2-f696-43e4-a0c9-c72bac59be25-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp\" (UID: \"4b7900e2-f696-43e4-a0c9-c72bac59be25\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:27.353378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.349992 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7900e2-f696-43e4-a0c9-c72bac59be25-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp\" (UID: \"4b7900e2-f696-43e4-a0c9-c72bac59be25\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:27.353378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.350156 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b7900e2-f696-43e4-a0c9-c72bac59be25-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp\" (UID: \"4b7900e2-f696-43e4-a0c9-c72bac59be25\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:27.353378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.350315 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwqx5\" (UniqueName: \"kubernetes.io/projected/4b7900e2-f696-43e4-a0c9-c72bac59be25-kube-api-access-mwqx5\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp\" (UID: \"4b7900e2-f696-43e4-a0c9-c72bac59be25\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:27.353378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.352390 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7900e2-f696-43e4-a0c9-c72bac59be25-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp\" (UID: \"4b7900e2-f696-43e4-a0c9-c72bac59be25\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:27.353378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.352936 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b7900e2-f696-43e4-a0c9-c72bac59be25-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp\" (UID: \"4b7900e2-f696-43e4-a0c9-c72bac59be25\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:27.362450 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.362420 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b7900e2-f696-43e4-a0c9-c72bac59be25-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp\" (UID: \"4b7900e2-f696-43e4-a0c9-c72bac59be25\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:27.371938 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.371919 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwqx5\" (UniqueName: \"kubernetes.io/projected/4b7900e2-f696-43e4-a0c9-c72bac59be25-kube-api-access-mwqx5\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp\" (UID: \"4b7900e2-f696-43e4-a0c9-c72bac59be25\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:27.377368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.374036 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7"] Oct 30 09:53:27.377368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.374208 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" Oct 30 09:53:27.404049 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.404015 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:53:27.404880 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.404852 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:53:27.408082 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.407310 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:27.584536 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.584472 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96b0fd9d-jkgm7\" (UID: \"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" Oct 30 09:53:27.584964 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.584681 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72525\" (UniqueName: \"kubernetes.io/projected/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-kube-api-access-72525\") pod \"maas-api-revoke-sub-96b0fd9d-jkgm7\" (UID: \"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" Oct 30 09:53:27.584964 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.584899 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96b0fd9d-jkgm7\" (UID: \"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" Oct 30 09:53:27.688233 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.688200 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72525\" (UniqueName: \"kubernetes.io/projected/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-kube-api-access-72525\") pod \"maas-api-revoke-sub-96b0fd9d-jkgm7\" (UID: \"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" Oct 30 09:53:27.688386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.688258 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96b0fd9d-jkgm7\" (UID: \"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" Oct 30 09:53:27.688521 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.688501 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96b0fd9d-jkgm7\" (UID: \"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" Oct 30 09:53:27.689067 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.689037 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96b0fd9d-jkgm7\" (UID: \"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" Oct 30 09:53:27.691412 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.691390 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96b0fd9d-jkgm7\" (UID: \"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" Oct 30 09:53:27.710316 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.710274 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72525\" (UniqueName: \"kubernetes.io/projected/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-kube-api-access-72525\") pod \"maas-api-revoke-sub-96b0fd9d-jkgm7\" (UID: \"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" Oct 30 09:53:27.883636 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.882849 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" Oct 30 09:53:27.907844 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:53:27.907784 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7900e2_f696_43e4_a0c9_c72bac59be25.slice/crio-e0a60080e5f7bb98b23520e5d6dbd54e90353d56b5b6bd67bf76b53b5fd66cd8 WatchSource:0}: Error finding container e0a60080e5f7bb98b23520e5d6dbd54e90353d56b5b6bd67bf76b53b5fd66cd8: Status 404 returned error can't find the container with id e0a60080e5f7bb98b23520e5d6dbd54e90353d56b5b6bd67bf76b53b5fd66cd8 Oct 30 09:53:27.908505 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:27.908479 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp"] Oct 30 09:53:28.080312 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.078949 5938 patch_prober.go:29] interesting pod/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 09:53:28.080312 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.078994 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" podUID="df118d0b-7613-4edf-aaa4-af1944531fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 09:53:28.083236 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.083199 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" event={"ID":"4b7900e2-f696-43e4-a0c9-c72bac59be25","Type":"ContainerStarted","Data":"7daf74ae85f1a1baa49d96b45b30da26aaa976041f9f7f328d773a9a37dc7b94"} Oct 30 09:53:28.083392 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.083272 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" event={"ID":"4b7900e2-f696-43e4-a0c9-c72bac59be25","Type":"ContainerStarted","Data":"e0a60080e5f7bb98b23520e5d6dbd54e90353d56b5b6bd67bf76b53b5fd66cd8"} Oct 30 09:53:28.084512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.084474 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:28.388856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.388819 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452"] Oct 30 09:53:28.394742 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.394645 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" podStartSLOduration=1.394626929 podStartE2EDuration="1.394626929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:28.143872047 +0000 UTC m=+3526.006577868" watchObservedRunningTime="2026-10-30 09:53:28.394626929 +0000 UTC m=+3526.257332751" Oct 30 09:53:28.399072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.399048 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:28.405123 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.405097 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452"] Oct 30 09:53:28.416166 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.416143 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3d9b36\"" Oct 30 09:53:28.416879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.416861 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"agc-cred-cab5f6d18ff4\"" Oct 30 09:53:28.416958 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.416930 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"kube-root-ca.crt\"" Oct 30 09:53:28.417118 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.417101 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"payload-processing-external-model-e2e--dd1a53db-dockercfg-p4mth\"" Oct 30 09:53:28.458896 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.458790 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7"] Oct 30 09:53:28.459952 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:53:28.459926 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5faef7a2_55ae_4c8e_9787_43dfa5a1b67a.slice/crio-f3af1595f163de31b405b583ff32817cae5314c6dc1d59219ed7b92eabfa4659 WatchSource:0}: Error finding container f3af1595f163de31b405b583ff32817cae5314c6dc1d59219ed7b92eabfa4659: Status 404 returned error can't find the container with id f3af1595f163de31b405b583ff32817cae5314c6dc1d59219ed7b92eabfa4659 Oct 30 09:53:28.516620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.516599 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"routing-overlay\"" Oct 30 09:53:28.516710 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.516625 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"openshift-service-ca.crt\"" Oct 30 09:53:28.545645 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.545610 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645092e4-7212-451f-bbcd-70fe8ff21fa9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452\" (UID: \"645092e4-7212-451f-bbcd-70fe8ff21fa9\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:28.545796 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.545758 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/645092e4-7212-451f-bbcd-70fe8ff21fa9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452\" (UID: \"645092e4-7212-451f-bbcd-70fe8ff21fa9\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:28.545876 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.545856 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/645092e4-7212-451f-bbcd-70fe8ff21fa9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452\" (UID: \"645092e4-7212-451f-bbcd-70fe8ff21fa9\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:28.545980 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.545962 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76rg6\" (UniqueName: \"kubernetes.io/projected/645092e4-7212-451f-bbcd-70fe8ff21fa9-kube-api-access-76rg6\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452\" (UID: \"645092e4-7212-451f-bbcd-70fe8ff21fa9\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:28.647947 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.647869 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645092e4-7212-451f-bbcd-70fe8ff21fa9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452\" (UID: \"645092e4-7212-451f-bbcd-70fe8ff21fa9\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:28.648124 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.648029 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/645092e4-7212-451f-bbcd-70fe8ff21fa9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452\" (UID: \"645092e4-7212-451f-bbcd-70fe8ff21fa9\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:28.648207 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.648133 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/645092e4-7212-451f-bbcd-70fe8ff21fa9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452\" (UID: \"645092e4-7212-451f-bbcd-70fe8ff21fa9\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:28.648270 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.648248 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76rg6\" (UniqueName: \"kubernetes.io/projected/645092e4-7212-451f-bbcd-70fe8ff21fa9-kube-api-access-76rg6\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452\" (UID: \"645092e4-7212-451f-bbcd-70fe8ff21fa9\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:28.648982 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.648789 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645092e4-7212-451f-bbcd-70fe8ff21fa9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452\" (UID: \"645092e4-7212-451f-bbcd-70fe8ff21fa9\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:28.649422 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.648996 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/645092e4-7212-451f-bbcd-70fe8ff21fa9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452\" (UID: \"645092e4-7212-451f-bbcd-70fe8ff21fa9\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:28.651256 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.651239 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/645092e4-7212-451f-bbcd-70fe8ff21fa9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452\" (UID: \"645092e4-7212-451f-bbcd-70fe8ff21fa9\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:28.669325 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.669303 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76rg6\" (UniqueName: \"kubernetes.io/projected/645092e4-7212-451f-bbcd-70fe8ff21fa9-kube-api-access-76rg6\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452\" (UID: \"645092e4-7212-451f-bbcd-70fe8ff21fa9\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:28.745612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:28.741865 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:29.005501 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.005468 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888"] Oct 30 09:53:29.011289 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.011223 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" Oct 30 09:53:29.056235 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.056191 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888"] Oct 30 09:53:29.149638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.147144 5938 patch_prober.go:29] interesting pod/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 09:53:29.149638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.147200 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" podUID="df118d0b-7613-4edf-aaa4-af1944531fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 09:53:29.172754 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.172705 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb070777-g4888\" (UID: \"52ab54ca-24c3-4bd3-a464-0d1a23bee04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" Oct 30 09:53:29.172924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.172796 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vd5j\" (UniqueName: \"kubernetes.io/projected/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-kube-api-access-5vd5j\") pod \"maas-api-revoke-sub-eb070777-g4888\" (UID: \"52ab54ca-24c3-4bd3-a464-0d1a23bee04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" Oct 30 09:53:29.172924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.172842 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb070777-g4888\" (UID: \"52ab54ca-24c3-4bd3-a464-0d1a23bee04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" Oct 30 09:53:29.177845 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.177793 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" event={"ID":"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a","Type":"ContainerStarted","Data":"f1911f620a59761db1f89d10085c520dad2813c9af54ddd9dc1818579c4fc4b6"} Oct 30 09:53:29.178001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.177866 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" event={"ID":"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a","Type":"ContainerStarted","Data":"f3af1595f163de31b405b583ff32817cae5314c6dc1d59219ed7b92eabfa4659"} Oct 30 09:53:29.183926 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.183907 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn"] Oct 30 09:53:29.193670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.190831 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn"] Oct 30 09:53:29.193670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.190957 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:29.253612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.253250 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" podStartSLOduration=2.253231803 podStartE2EDuration="2.253231803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:29.250542904 +0000 UTC m=+3527.113248738" watchObservedRunningTime="2026-10-30 09:53:29.253231803 +0000 UTC m=+3527.115937625" Oct 30 09:53:29.274922 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.274704 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb070777-g4888\" (UID: \"52ab54ca-24c3-4bd3-a464-0d1a23bee04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" Oct 30 09:53:29.275214 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.275191 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb070777-g4888\" (UID: \"52ab54ca-24c3-4bd3-a464-0d1a23bee04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" Oct 30 09:53:29.275317 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.275279 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vd5j\" (UniqueName: \"kubernetes.io/projected/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-kube-api-access-5vd5j\") pod \"maas-api-revoke-sub-eb070777-g4888\" (UID: \"52ab54ca-24c3-4bd3-a464-0d1a23bee04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" Oct 30 09:53:29.277247 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.276401 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb070777-g4888\" (UID: \"52ab54ca-24c3-4bd3-a464-0d1a23bee04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" Oct 30 09:53:29.278747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.278729 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb070777-g4888\" (UID: \"52ab54ca-24c3-4bd3-a464-0d1a23bee04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" Oct 30 09:53:29.310836 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.310025 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vd5j\" (UniqueName: \"kubernetes.io/projected/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-kube-api-access-5vd5j\") pod \"maas-api-revoke-sub-eb070777-g4888\" (UID: \"52ab54ca-24c3-4bd3-a464-0d1a23bee04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" Oct 30 09:53:29.374957 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.372048 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" Oct 30 09:53:29.374957 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.373257 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452"] Oct 30 09:53:29.380042 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.379395 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn\" (UID: \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:29.380042 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.379432 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnmjb\" (UniqueName: \"kubernetes.io/projected/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-kube-api-access-wnmjb\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn\" (UID: \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:29.380042 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.379559 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn\" (UID: \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:29.380042 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.379797 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn\" (UID: \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:29.387651 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:53:29.386108 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645092e4_7212_451f_bbcd_70fe8ff21fa9.slice/crio-8663ff8bdda33892aad461691862821a5543ceea6678dde30180a150ebb14eba WatchSource:0}: Error finding container 8663ff8bdda33892aad461691862821a5543ceea6678dde30180a150ebb14eba: Status 404 returned error can't find the container with id 8663ff8bdda33892aad461691862821a5543ceea6678dde30180a150ebb14eba Oct 30 09:53:29.481769 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.481733 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn\" (UID: \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:29.481925 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.481899 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn\" (UID: \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:29.482014 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.481927 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnmjb\" (UniqueName: \"kubernetes.io/projected/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-kube-api-access-wnmjb\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn\" (UID: \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:29.482397 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.482063 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn\" (UID: \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:29.482544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.482451 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn\" (UID: \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:29.483037 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.483016 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn\" (UID: \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:29.484686 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.484665 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn\" (UID: \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:29.510118 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.510090 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnmjb\" (UniqueName: \"kubernetes.io/projected/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-kube-api-access-wnmjb\") pod \"payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn\" (UID: \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\") " pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:29.519517 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.519491 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:29.752536 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.752502 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888"] Oct 30 09:53:29.753926 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:53:29.753901 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ab54ca_24c3_4bd3_a464_0d1a23bee04a.slice/crio-592c6a5a1a69d55dfe603074bffd9fa904d89a48ab93a76f687b8dbcca553e56 WatchSource:0}: Error finding container 592c6a5a1a69d55dfe603074bffd9fa904d89a48ab93a76f687b8dbcca553e56: Status 404 returned error can't find the container with id 592c6a5a1a69d55dfe603074bffd9fa904d89a48ab93a76f687b8dbcca553e56 Oct 30 09:53:29.871844 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:29.871817 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn"] Oct 30 09:53:29.872443 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:53:29.872419 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1aa1ff0_c8cf_4289_b9cd_f792be68a9c1.slice/crio-2aaa5d35a3c32a84fbce520fbe10e14881cff9b6113a5fb557a2f317967888d8 WatchSource:0}: Error finding container 2aaa5d35a3c32a84fbce520fbe10e14881cff9b6113a5fb557a2f317967888d8: Status 404 returned error can't find the container with id 2aaa5d35a3c32a84fbce520fbe10e14881cff9b6113a5fb557a2f317967888d8 Oct 30 09:53:30.082809 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:30.082727 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:30.083222 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:30.083203 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:30.083913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:30.083897 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:53:30.191957 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:30.191922 5938 generic.go:356] "Generic (PLEG): container finished" podID="5faef7a2-55ae-4c8e-9787-43dfa5a1b67a" containerID="f1911f620a59761db1f89d10085c520dad2813c9af54ddd9dc1818579c4fc4b6" exitCode=0 Oct 30 09:53:30.192465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:30.192052 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" event={"ID":"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a","Type":"ContainerDied","Data":"f1911f620a59761db1f89d10085c520dad2813c9af54ddd9dc1818579c4fc4b6"} Oct 30 09:53:30.194182 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:30.194155 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" event={"ID":"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1","Type":"ContainerStarted","Data":"aa89e993c654532f1ce6fb4cdf6bfe2e8607bbef102b03d68f2d5c6a4ddfa9b9"} Oct 30 09:53:30.194333 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:30.194206 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:30.194333 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:30.194234 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" event={"ID":"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1","Type":"ContainerStarted","Data":"2aaa5d35a3c32a84fbce520fbe10e14881cff9b6113a5fb557a2f317967888d8"} Oct 30 09:53:30.196654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:30.196625 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" event={"ID":"645092e4-7212-451f-bbcd-70fe8ff21fa9","Type":"ContainerStarted","Data":"5ff290fded5deb6a8869a97419120e73df68f4004ba86ca7036414d85d39ab86"} Oct 30 09:53:30.196755 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:30.196680 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" event={"ID":"645092e4-7212-451f-bbcd-70fe8ff21fa9","Type":"ContainerStarted","Data":"8663ff8bdda33892aad461691862821a5543ceea6678dde30180a150ebb14eba"} Oct 30 09:53:30.197445 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:30.197423 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:30.210683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:30.209453 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" event={"ID":"52ab54ca-24c3-4bd3-a464-0d1a23bee04a","Type":"ContainerStarted","Data":"ce35963cc91e32b2c69b49e87f48a92717c2f983753a629309a870080d3b0ba5"} Oct 30 09:53:30.210683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:30.209494 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" event={"ID":"52ab54ca-24c3-4bd3-a464-0d1a23bee04a","Type":"ContainerStarted","Data":"592c6a5a1a69d55dfe603074bffd9fa904d89a48ab93a76f687b8dbcca553e56"} Oct 30 09:53:30.278533 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:30.278470 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" podStartSLOduration=1.278456764 podStartE2EDuration="1.278456764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:30.275260251 +0000 UTC m=+3528.137966072" watchObservedRunningTime="2026-10-30 09:53:30.278456764 +0000 UTC m=+3528.141162585" Oct 30 09:53:30.321120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:30.321074 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" podStartSLOduration=2.321057596 podStartE2EDuration="2.321057596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:30.320410381 +0000 UTC m=+3528.183116205" watchObservedRunningTime="2026-10-30 09:53:30.321057596 +0000 UTC m=+3528.183763418" Oct 30 09:53:30.360931 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:30.360842 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" podStartSLOduration=2.360828587 podStartE2EDuration="2.360828587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:30.360297543 +0000 UTC m=+3528.223003365" watchObservedRunningTime="2026-10-30 09:53:30.360828587 +0000 UTC m=+3528.223534408" Oct 30 09:53:31.229568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:31.227788 5938 generic.go:356] "Generic (PLEG): container finished" podID="52ab54ca-24c3-4bd3-a464-0d1a23bee04a" containerID="ce35963cc91e32b2c69b49e87f48a92717c2f983753a629309a870080d3b0ba5" exitCode=0 Oct 30 09:53:31.229568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:31.229302 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" event={"ID":"52ab54ca-24c3-4bd3-a464-0d1a23bee04a","Type":"ContainerDied","Data":"ce35963cc91e32b2c69b49e87f48a92717c2f983753a629309a870080d3b0ba5"} Oct 30 09:53:31.405059 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:31.405031 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" Oct 30 09:53:31.435327 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:31.435298 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-kube-api-access-72525\" (UniqueName: \"kubernetes.io/projected/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-kube-api-access-72525\") pod \"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a\" (UID: \"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a\") " Oct 30 09:53:31.435451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:31.435440 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-maas-api-openshift-ca\") pod \"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a\" (UID: \"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a\") " Oct 30 09:53:31.435495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:31.435467 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-maas-api-xks-ca\") pod \"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a\" (UID: \"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a\") " Oct 30 09:53:31.435846 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:31.435816 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-maas-api-openshift-ca" pod "5faef7a2-55ae-4c8e-9787-43dfa5a1b67a" (UID: "5faef7a2-55ae-4c8e-9787-43dfa5a1b67a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:31.436202 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:31.436145 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:31.437541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:31.437523 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-maas-api-xks-ca" pod "5faef7a2-55ae-4c8e-9787-43dfa5a1b67a" (UID: "5faef7a2-55ae-4c8e-9787-43dfa5a1b67a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:53:31.437671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:31.437657 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-kube-api-access-72525" pod "5faef7a2-55ae-4c8e-9787-43dfa5a1b67a" (UID: "5faef7a2-55ae-4c8e-9787-43dfa5a1b67a"). InnerVolumeSpecName "kube-api-access-72525". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:31.537223 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:31.537149 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72525\" (UniqueName: \"kubernetes.io/projected/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-kube-api-access-72525\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:31.537223 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:31.537178 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:32.237673 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:32.237635 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" event={"ID":"5faef7a2-55ae-4c8e-9787-43dfa5a1b67a","Type":"ContainerDied","Data":"f3af1595f163de31b405b583ff32817cae5314c6dc1d59219ed7b92eabfa4659"} Oct 30 09:53:32.238142 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:32.237697 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3af1595f163de31b405b583ff32817cae5314c6dc1d59219ed7b92eabfa4659" Oct 30 09:53:32.238142 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:32.237717 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7" Oct 30 09:53:32.408218 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:32.408191 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" Oct 30 09:53:32.450891 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:32.450861 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-maas-api-openshift-ca\") pod \"52ab54ca-24c3-4bd3-a464-0d1a23bee04a\" (UID: \"52ab54ca-24c3-4bd3-a464-0d1a23bee04a\") " Oct 30 09:53:32.451041 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:32.450908 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-kube-api-access-5vd5j\" (UniqueName: \"kubernetes.io/projected/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-kube-api-access-5vd5j\") pod \"52ab54ca-24c3-4bd3-a464-0d1a23bee04a\" (UID: \"52ab54ca-24c3-4bd3-a464-0d1a23bee04a\") " Oct 30 09:53:32.451041 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:32.450988 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-maas-api-xks-ca\") pod \"52ab54ca-24c3-4bd3-a464-0d1a23bee04a\" (UID: \"52ab54ca-24c3-4bd3-a464-0d1a23bee04a\") " Oct 30 09:53:32.451279 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:32.451235 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-maas-api-openshift-ca" pod "52ab54ca-24c3-4bd3-a464-0d1a23bee04a" (UID: "52ab54ca-24c3-4bd3-a464-0d1a23bee04a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:32.451876 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:32.451808 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:32.453256 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:32.453225 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-kube-api-access-5vd5j" pod "52ab54ca-24c3-4bd3-a464-0d1a23bee04a" (UID: "52ab54ca-24c3-4bd3-a464-0d1a23bee04a"). InnerVolumeSpecName "kube-api-access-5vd5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:32.453339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:32.453280 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-maas-api-xks-ca" pod "52ab54ca-24c3-4bd3-a464-0d1a23bee04a" (UID: "52ab54ca-24c3-4bd3-a464-0d1a23bee04a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:53:32.553373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:32.553306 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:32.553373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:32.553339 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vd5j\" (UniqueName: \"kubernetes.io/projected/52ab54ca-24c3-4bd3-a464-0d1a23bee04a-kube-api-access-5vd5j\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:33.246799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:33.246769 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" Oct 30 09:53:33.247215 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:33.246795 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888" event={"ID":"52ab54ca-24c3-4bd3-a464-0d1a23bee04a","Type":"ContainerDied","Data":"592c6a5a1a69d55dfe603074bffd9fa904d89a48ab93a76f687b8dbcca553e56"} Oct 30 09:53:33.247215 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:33.246847 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="592c6a5a1a69d55dfe603074bffd9fa904d89a48ab93a76f687b8dbcca553e56" Oct 30 09:53:38.046342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.046304 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg"] Oct 30 09:53:38.048876 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.048848 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5faef7a2-55ae-4c8e-9787-43dfa5a1b67a" containerName="revoke-keys" Oct 30 09:53:38.049039 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.049025 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5faef7a2-55ae-4c8e-9787-43dfa5a1b67a" containerName="revoke-keys" Oct 30 09:53:38.049247 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.049235 5938 cpu_manager.go:414] "containerMap: removing container" podUID="52ab54ca-24c3-4bd3-a464-0d1a23bee04a" containerName="revoke-keys" Oct 30 09:53:38.049337 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.049326 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ab54ca-24c3-4bd3-a464-0d1a23bee04a" containerName="revoke-keys" Oct 30 09:53:38.050145 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.050123 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ab54ca-24c3-4bd3-a464-0d1a23bee04a" containerName="revoke-keys" Oct 30 09:53:38.050304 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.050292 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5faef7a2-55ae-4c8e-9787-43dfa5a1b67a" containerName="revoke-keys" Oct 30 09:53:38.059418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.059396 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" Oct 30 09:53:38.062907 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.062884 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg"] Oct 30 09:53:38.075908 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.075882 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:53:38.089765 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.089749 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:53:38.128028 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.127963 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6102d5ea-7a81-45a0-81b0-afead85c8871-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd0e6e2-kzxhg\" (UID: \"6102d5ea-7a81-45a0-81b0-afead85c8871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" Oct 30 09:53:38.128196 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.128083 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6102d5ea-7a81-45a0-81b0-afead85c8871-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd0e6e2-kzxhg\" (UID: \"6102d5ea-7a81-45a0-81b0-afead85c8871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" Oct 30 09:53:38.128251 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.128213 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkrhj\" (UniqueName: \"kubernetes.io/projected/6102d5ea-7a81-45a0-81b0-afead85c8871-kube-api-access-mkrhj\") pod \"maas-api-revoke-sub-cbd0e6e2-kzxhg\" (UID: \"6102d5ea-7a81-45a0-81b0-afead85c8871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" Oct 30 09:53:38.229206 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.229172 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6102d5ea-7a81-45a0-81b0-afead85c8871-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd0e6e2-kzxhg\" (UID: \"6102d5ea-7a81-45a0-81b0-afead85c8871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" Oct 30 09:53:38.229432 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.229262 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6102d5ea-7a81-45a0-81b0-afead85c8871-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd0e6e2-kzxhg\" (UID: \"6102d5ea-7a81-45a0-81b0-afead85c8871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" Oct 30 09:53:38.229432 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.229348 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkrhj\" (UniqueName: \"kubernetes.io/projected/6102d5ea-7a81-45a0-81b0-afead85c8871-kube-api-access-mkrhj\") pod \"maas-api-revoke-sub-cbd0e6e2-kzxhg\" (UID: \"6102d5ea-7a81-45a0-81b0-afead85c8871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" Oct 30 09:53:38.229989 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.229967 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6102d5ea-7a81-45a0-81b0-afead85c8871-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd0e6e2-kzxhg\" (UID: \"6102d5ea-7a81-45a0-81b0-afead85c8871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" Oct 30 09:53:38.232082 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.232058 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6102d5ea-7a81-45a0-81b0-afead85c8871-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd0e6e2-kzxhg\" (UID: \"6102d5ea-7a81-45a0-81b0-afead85c8871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" Oct 30 09:53:38.250646 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.250618 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkrhj\" (UniqueName: \"kubernetes.io/projected/6102d5ea-7a81-45a0-81b0-afead85c8871-kube-api-access-mkrhj\") pod \"maas-api-revoke-sub-cbd0e6e2-kzxhg\" (UID: \"6102d5ea-7a81-45a0-81b0-afead85c8871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" Oct 30 09:53:38.379096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.379069 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" Oct 30 09:53:38.689771 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:38.689667 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg"] Oct 30 09:53:39.305172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:39.303206 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" event={"ID":"6102d5ea-7a81-45a0-81b0-afead85c8871","Type":"ContainerStarted","Data":"3ffd0b2714207ba494a6cca997722003c0aba53270dbad515f6aab8c2a502128"} Oct 30 09:53:39.305172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:39.303280 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" event={"ID":"6102d5ea-7a81-45a0-81b0-afead85c8871","Type":"ContainerStarted","Data":"c524c6031e7cc93cefdfef2d7940690587010a9014baa01d03f4d9c39f6bc97b"} Oct 30 09:53:39.350360 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:39.350299 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" podStartSLOduration=1.350284639 podStartE2EDuration="1.350284639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:39.346286522 +0000 UTC m=+3537.208992354" watchObservedRunningTime="2026-10-30 09:53:39.350284639 +0000 UTC m=+3537.212990460" Oct 30 09:53:39.754387 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:39.754353 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f"] Oct 30 09:53:39.760054 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:39.760028 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" Oct 30 09:53:39.776380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:39.776328 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f"] Oct 30 09:53:39.854428 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:39.854357 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4dacfa7-b373-403f-8881-537132dd2997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-891b53c6-qbk8f\" (UID: \"b4dacfa7-b373-403f-8881-537132dd2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" Oct 30 09:53:39.854579 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:39.854422 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4dacfa7-b373-403f-8881-537132dd2997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-891b53c6-qbk8f\" (UID: \"b4dacfa7-b373-403f-8881-537132dd2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" Oct 30 09:53:39.854579 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:39.854516 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mkt7\" (UniqueName: \"kubernetes.io/projected/b4dacfa7-b373-403f-8881-537132dd2997-kube-api-access-4mkt7\") pod \"maas-api-revoke-sub-891b53c6-qbk8f\" (UID: \"b4dacfa7-b373-403f-8881-537132dd2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" Oct 30 09:53:39.955854 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:39.955823 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4dacfa7-b373-403f-8881-537132dd2997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-891b53c6-qbk8f\" (UID: \"b4dacfa7-b373-403f-8881-537132dd2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" Oct 30 09:53:39.956038 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:39.955892 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4dacfa7-b373-403f-8881-537132dd2997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-891b53c6-qbk8f\" (UID: \"b4dacfa7-b373-403f-8881-537132dd2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" Oct 30 09:53:39.956038 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:39.955962 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mkt7\" (UniqueName: \"kubernetes.io/projected/b4dacfa7-b373-403f-8881-537132dd2997-kube-api-access-4mkt7\") pod \"maas-api-revoke-sub-891b53c6-qbk8f\" (UID: \"b4dacfa7-b373-403f-8881-537132dd2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" Oct 30 09:53:39.956618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:39.956560 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4dacfa7-b373-403f-8881-537132dd2997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-891b53c6-qbk8f\" (UID: \"b4dacfa7-b373-403f-8881-537132dd2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" Oct 30 09:53:39.958204 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:39.958182 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4dacfa7-b373-403f-8881-537132dd2997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-891b53c6-qbk8f\" (UID: \"b4dacfa7-b373-403f-8881-537132dd2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" Oct 30 09:53:39.976059 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:39.976033 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mkt7\" (UniqueName: \"kubernetes.io/projected/b4dacfa7-b373-403f-8881-537132dd2997-kube-api-access-4mkt7\") pod \"maas-api-revoke-sub-891b53c6-qbk8f\" (UID: \"b4dacfa7-b373-403f-8881-537132dd2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" Oct 30 09:53:40.077410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:40.077372 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" Oct 30 09:53:40.213304 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:40.213273 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:40.313981 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:40.313948 5938 generic.go:356] "Generic (PLEG): container finished" podID="6102d5ea-7a81-45a0-81b0-afead85c8871" containerID="3ffd0b2714207ba494a6cca997722003c0aba53270dbad515f6aab8c2a502128" exitCode=0 Oct 30 09:53:40.314447 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:40.314041 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" event={"ID":"6102d5ea-7a81-45a0-81b0-afead85c8871","Type":"ContainerDied","Data":"3ffd0b2714207ba494a6cca997722003c0aba53270dbad515f6aab8c2a502128"} Oct 30 09:53:40.421573 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:53:40.421541 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4dacfa7_b373_403f_8881_537132dd2997.slice/crio-3b57bcdb796f15a664222181a09c54136ac1da794a7cba654230a16fcb82ed2b WatchSource:0}: Error finding container 3b57bcdb796f15a664222181a09c54136ac1da794a7cba654230a16fcb82ed2b: Status 404 returned error can't find the container with id 3b57bcdb796f15a664222181a09c54136ac1da794a7cba654230a16fcb82ed2b Oct 30 09:53:40.424167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:40.424144 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f"] Oct 30 09:53:41.235619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:41.235548 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:41.309625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:41.309552 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl"] Oct 30 09:53:41.327018 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:41.326970 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" event={"ID":"b4dacfa7-b373-403f-8881-537132dd2997","Type":"ContainerStarted","Data":"57aacc5a6e3b004cdcaff613ea471059d2e699282b881264b496937e15c7d380"} Oct 30 09:53:41.327428 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:41.327046 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" event={"ID":"b4dacfa7-b373-403f-8881-537132dd2997","Type":"ContainerStarted","Data":"3b57bcdb796f15a664222181a09c54136ac1da794a7cba654230a16fcb82ed2b"} Oct 30 09:53:41.397400 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:41.397349 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" podStartSLOduration=2.397330865 podStartE2EDuration="2.397330865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:41.393113143 +0000 UTC m=+3539.255818976" watchObservedRunningTime="2026-10-30 09:53:41.397330865 +0000 UTC m=+3539.260036687" Oct 30 09:53:41.508505 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:41.508453 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" Oct 30 09:53:41.583838 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:41.583812 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6102d5ea-7a81-45a0-81b0-afead85c8871-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6102d5ea-7a81-45a0-81b0-afead85c8871-maas-api-openshift-ca\") pod \"6102d5ea-7a81-45a0-81b0-afead85c8871\" (UID: \"6102d5ea-7a81-45a0-81b0-afead85c8871\") " Oct 30 09:53:41.584004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:41.583852 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6102d5ea-7a81-45a0-81b0-afead85c8871-kube-api-access-mkrhj\" (UniqueName: \"kubernetes.io/projected/6102d5ea-7a81-45a0-81b0-afead85c8871-kube-api-access-mkrhj\") pod \"6102d5ea-7a81-45a0-81b0-afead85c8871\" (UID: \"6102d5ea-7a81-45a0-81b0-afead85c8871\") " Oct 30 09:53:41.584004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:41.583987 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6102d5ea-7a81-45a0-81b0-afead85c8871-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6102d5ea-7a81-45a0-81b0-afead85c8871-maas-api-xks-ca\") pod \"6102d5ea-7a81-45a0-81b0-afead85c8871\" (UID: \"6102d5ea-7a81-45a0-81b0-afead85c8871\") " Oct 30 09:53:41.584224 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:41.584197 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6102d5ea-7a81-45a0-81b0-afead85c8871-maas-api-openshift-ca" pod "6102d5ea-7a81-45a0-81b0-afead85c8871" (UID: "6102d5ea-7a81-45a0-81b0-afead85c8871"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:41.584675 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:41.584654 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6102d5ea-7a81-45a0-81b0-afead85c8871-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:41.586236 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:41.586210 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6102d5ea-7a81-45a0-81b0-afead85c8871-kube-api-access-mkrhj" pod "6102d5ea-7a81-45a0-81b0-afead85c8871" (UID: "6102d5ea-7a81-45a0-81b0-afead85c8871"). InnerVolumeSpecName "kube-api-access-mkrhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:41.586725 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:41.586691 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6102d5ea-7a81-45a0-81b0-afead85c8871-maas-api-xks-ca" pod "6102d5ea-7a81-45a0-81b0-afead85c8871" (UID: "6102d5ea-7a81-45a0-81b0-afead85c8871"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:53:41.686348 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:41.686321 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkrhj\" (UniqueName: \"kubernetes.io/projected/6102d5ea-7a81-45a0-81b0-afead85c8871-kube-api-access-mkrhj\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:41.686348 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:41.686349 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6102d5ea-7a81-45a0-81b0-afead85c8871-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:42.241126 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:42.241102 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:42.341747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:42.341709 5938 generic.go:356] "Generic (PLEG): container finished" podID="b4dacfa7-b373-403f-8881-537132dd2997" containerID="57aacc5a6e3b004cdcaff613ea471059d2e699282b881264b496937e15c7d380" exitCode=0 Oct 30 09:53:42.342207 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:42.341782 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" event={"ID":"b4dacfa7-b373-403f-8881-537132dd2997","Type":"ContainerDied","Data":"57aacc5a6e3b004cdcaff613ea471059d2e699282b881264b496937e15c7d380"} Oct 30 09:53:42.343678 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:42.343646 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" event={"ID":"6102d5ea-7a81-45a0-81b0-afead85c8871","Type":"ContainerDied","Data":"c524c6031e7cc93cefdfef2d7940690587010a9014baa01d03f4d9c39f6bc97b"} Oct 30 09:53:42.343803 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:42.343675 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg" Oct 30 09:53:42.343803 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:42.343689 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c524c6031e7cc93cefdfef2d7940690587010a9014baa01d03f4d9c39f6bc97b" Oct 30 09:53:42.880103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:42.880071 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz"] Oct 30 09:53:43.234828 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:43.234754 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f"] Oct 30 09:53:43.239813 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:43.239788 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-881f83-cmg5f"] Oct 30 09:53:43.536385 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:43.536355 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" Oct 30 09:53:43.616385 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:43.616352 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4dacfa7-b373-403f-8881-537132dd2997-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4dacfa7-b373-403f-8881-537132dd2997-maas-api-openshift-ca\") pod \"b4dacfa7-b373-403f-8881-537132dd2997\" (UID: \"b4dacfa7-b373-403f-8881-537132dd2997\") " Oct 30 09:53:43.616560 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:43.616399 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4dacfa7-b373-403f-8881-537132dd2997-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4dacfa7-b373-403f-8881-537132dd2997-maas-api-xks-ca\") pod \"b4dacfa7-b373-403f-8881-537132dd2997\" (UID: \"b4dacfa7-b373-403f-8881-537132dd2997\") " Oct 30 09:53:43.616560 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:43.616450 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4dacfa7-b373-403f-8881-537132dd2997-kube-api-access-4mkt7\" (UniqueName: \"kubernetes.io/projected/b4dacfa7-b373-403f-8881-537132dd2997-kube-api-access-4mkt7\") pod \"b4dacfa7-b373-403f-8881-537132dd2997\" (UID: \"b4dacfa7-b373-403f-8881-537132dd2997\") " Oct 30 09:53:43.616782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:43.616759 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4dacfa7-b373-403f-8881-537132dd2997-maas-api-openshift-ca" pod "b4dacfa7-b373-403f-8881-537132dd2997" (UID: "b4dacfa7-b373-403f-8881-537132dd2997"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:43.617156 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:43.617125 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4dacfa7-b373-403f-8881-537132dd2997-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:43.618850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:43.618814 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4dacfa7-b373-403f-8881-537132dd2997-maas-api-xks-ca" pod "b4dacfa7-b373-403f-8881-537132dd2997" (UID: "b4dacfa7-b373-403f-8881-537132dd2997"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:53:43.619570 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:43.619540 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4dacfa7-b373-403f-8881-537132dd2997-kube-api-access-4mkt7" pod "b4dacfa7-b373-403f-8881-537132dd2997" (UID: "b4dacfa7-b373-403f-8881-537132dd2997"). InnerVolumeSpecName "kube-api-access-4mkt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:43.718896 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:43.718869 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4dacfa7-b373-403f-8881-537132dd2997-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:43.718896 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:43.718892 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mkt7\" (UniqueName: \"kubernetes.io/projected/b4dacfa7-b373-403f-8881-537132dd2997-kube-api-access-4mkt7\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:44.259184 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:44.259152 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58ee945-e9e0-4d97-84a2-b31da7fe494b" path="/var/lib/kubelet/pods/f58ee945-e9e0-4d97-84a2-b31da7fe494b/volumes" Oct 30 09:53:44.337478 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:44.337441 5938 scope.go:122] "RemoveContainer" containerID="a5a5c43c5bea3503c9ecd11ed6ee3f6f3bd3febba92c3f6c38e5d77b91bc1443" Oct 30 09:53:44.353269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:44.353102 5938 scope.go:122] "RemoveContainer" containerID="5bf60015c1f14270793ea54044a06de2f60010269267f27526337639cdd8a385" Oct 30 09:53:44.361556 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:44.361532 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" Oct 30 09:53:44.361693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:44.361556 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f" event={"ID":"b4dacfa7-b373-403f-8881-537132dd2997","Type":"ContainerDied","Data":"3b57bcdb796f15a664222181a09c54136ac1da794a7cba654230a16fcb82ed2b"} Oct 30 09:53:44.361693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:44.361625 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b57bcdb796f15a664222181a09c54136ac1da794a7cba654230a16fcb82ed2b" Oct 30 09:53:45.811268 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:53:45.811215 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:53:46.310446 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:46.310409 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" podUID="befb486a-0145-4d7e-934b-6d4d01e5aa0f" containerName="payload-processing" containerID="cri-o://f6723b818a6e24a4bc5b7e27dfc7a26304f629536842bdaef33fe5758bec8a47" gracePeriod=25 Oct 30 09:53:46.601208 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:46.601061 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:53:46.764746 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:46.764704 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/befb486a-0145-4d7e-934b-6d4d01e5aa0f-kube-api-access-28fjp\" (UniqueName: \"kubernetes.io/projected/befb486a-0145-4d7e-934b-6d4d01e5aa0f-kube-api-access-28fjp\") pod \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\" (UID: \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\") " Oct 30 09:53:46.766131 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:46.766096 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/befb486a-0145-4d7e-934b-6d4d01e5aa0f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/befb486a-0145-4d7e-934b-6d4d01e5aa0f-routing-overlay-volume\") pod \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\" (UID: \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\") " Oct 30 09:53:46.766929 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:46.766909 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/befb486a-0145-4d7e-934b-6d4d01e5aa0f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/befb486a-0145-4d7e-934b-6d4d01e5aa0f-provider-credentials\") pod \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\" (UID: \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\") " Oct 30 09:53:46.767484 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:46.767466 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/befb486a-0145-4d7e-934b-6d4d01e5aa0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/befb486a-0145-4d7e-934b-6d4d01e5aa0f-plugins-config-volume\") pod \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\" (UID: \"befb486a-0145-4d7e-934b-6d4d01e5aa0f\") " Oct 30 09:53:46.768869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:46.766719 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/befb486a-0145-4d7e-934b-6d4d01e5aa0f-routing-overlay-volume" pod "befb486a-0145-4d7e-934b-6d4d01e5aa0f" (UID: "befb486a-0145-4d7e-934b-6d4d01e5aa0f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:46.769020 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:46.768021 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/befb486a-0145-4d7e-934b-6d4d01e5aa0f-plugins-config-volume" pod "befb486a-0145-4d7e-934b-6d4d01e5aa0f" (UID: "befb486a-0145-4d7e-934b-6d4d01e5aa0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:46.794625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:46.794300 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/befb486a-0145-4d7e-934b-6d4d01e5aa0f-provider-credentials" pod "befb486a-0145-4d7e-934b-6d4d01e5aa0f" (UID: "befb486a-0145-4d7e-934b-6d4d01e5aa0f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:46.795317 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:46.795277 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/befb486a-0145-4d7e-934b-6d4d01e5aa0f-kube-api-access-28fjp" pod "befb486a-0145-4d7e-934b-6d4d01e5aa0f" (UID: "befb486a-0145-4d7e-934b-6d4d01e5aa0f"). InnerVolumeSpecName "kube-api-access-28fjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:46.882615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:46.871883 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28fjp\" (UniqueName: \"kubernetes.io/projected/befb486a-0145-4d7e-934b-6d4d01e5aa0f-kube-api-access-28fjp\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:46.882615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:46.871919 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/befb486a-0145-4d7e-934b-6d4d01e5aa0f-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:46.882615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:46.871935 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/befb486a-0145-4d7e-934b-6d4d01e5aa0f-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:46.882615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:46.871952 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/befb486a-0145-4d7e-934b-6d4d01e5aa0f-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:47.397661 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:47.394337 5938 generic.go:356] "Generic (PLEG): container finished" podID="befb486a-0145-4d7e-934b-6d4d01e5aa0f" containerID="f6723b818a6e24a4bc5b7e27dfc7a26304f629536842bdaef33fe5758bec8a47" exitCode=0 Oct 30 09:53:47.397661 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:47.394438 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" event={"ID":"befb486a-0145-4d7e-934b-6d4d01e5aa0f","Type":"ContainerDied","Data":"f6723b818a6e24a4bc5b7e27dfc7a26304f629536842bdaef33fe5758bec8a47"} Oct 30 09:53:47.397661 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:47.394495 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" event={"ID":"befb486a-0145-4d7e-934b-6d4d01e5aa0f","Type":"ContainerDied","Data":"562ae5f1adde7deb1291cd368ab57b3dcde39ebaeabe23608838223b2abc973f"} Oct 30 09:53:47.397661 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:47.394524 5938 scope.go:122] "RemoveContainer" containerID="f6723b818a6e24a4bc5b7e27dfc7a26304f629536842bdaef33fe5758bec8a47" Oct 30 09:53:47.397661 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:47.394718 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl" Oct 30 09:53:47.427003 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:47.426978 5938 scope.go:122] "RemoveContainer" containerID="f6723b818a6e24a4bc5b7e27dfc7a26304f629536842bdaef33fe5758bec8a47" Oct 30 09:53:47.427440 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:53:47.427413 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6723b818a6e24a4bc5b7e27dfc7a26304f629536842bdaef33fe5758bec8a47\": container with ID starting with f6723b818a6e24a4bc5b7e27dfc7a26304f629536842bdaef33fe5758bec8a47 not found: ID does not exist" containerID="f6723b818a6e24a4bc5b7e27dfc7a26304f629536842bdaef33fe5758bec8a47" Oct 30 09:53:47.427553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:47.427451 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6723b818a6e24a4bc5b7e27dfc7a26304f629536842bdaef33fe5758bec8a47"} err="failed to get container status \"f6723b818a6e24a4bc5b7e27dfc7a26304f629536842bdaef33fe5758bec8a47\": rpc error: code = NotFound desc = could not find container \"f6723b818a6e24a4bc5b7e27dfc7a26304f629536842bdaef33fe5758bec8a47\": container with ID starting with f6723b818a6e24a4bc5b7e27dfc7a26304f629536842bdaef33fe5758bec8a47 not found: ID does not exist" Oct 30 09:53:47.468000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:47.467967 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl"] Oct 30 09:53:47.481673 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:47.481634 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-5d69whqrl"] Oct 30 09:53:47.881166 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:47.881130 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" podUID="f7e4d501-83df-4eee-9008-a1da59cd2ee1" containerName="payload-processing" containerID="cri-o://45af0ca963909d595ab1d6d2b2c8aa45a221eb58d6ac5e5eef9d5e310f0f312b" gracePeriod=25 Oct 30 09:53:47.895933 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:53:47.895908 5938 logging.go:55] [core] [Channel #6578 SubChannel #6579]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.215:50052", ServerName: "10.134.0.215:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.215:50052: connect: connection refused" Oct 30 09:53:48.414361 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.413841 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="befb486a-0145-4d7e-934b-6d4d01e5aa0f" path="/var/lib/kubelet/pods/befb486a-0145-4d7e-934b-6d4d01e5aa0f/volumes" Oct 30 09:53:48.425618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.417024 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:53:48.483612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.470778 5938 generic.go:356] "Generic (PLEG): container finished" podID="f7e4d501-83df-4eee-9008-a1da59cd2ee1" containerID="45af0ca963909d595ab1d6d2b2c8aa45a221eb58d6ac5e5eef9d5e310f0f312b" exitCode=0 Oct 30 09:53:48.483612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.470980 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" event={"ID":"f7e4d501-83df-4eee-9008-a1da59cd2ee1","Type":"ContainerDied","Data":"45af0ca963909d595ab1d6d2b2c8aa45a221eb58d6ac5e5eef9d5e310f0f312b"} Oct 30 09:53:48.483612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.471028 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" event={"ID":"f7e4d501-83df-4eee-9008-a1da59cd2ee1","Type":"ContainerDied","Data":"6258845e8f23134d33e95a5c12a0dc6627df8768598cab88785f414354ce955b"} Oct 30 09:53:48.483612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.471058 5938 scope.go:122] "RemoveContainer" containerID="45af0ca963909d595ab1d6d2b2c8aa45a221eb58d6ac5e5eef9d5e310f0f312b" Oct 30 09:53:48.483612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.471166 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" Oct 30 09:53:48.617490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.589508 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7e4d501-83df-4eee-9008-a1da59cd2ee1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7e4d501-83df-4eee-9008-a1da59cd2ee1-routing-overlay-volume\") pod \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\" (UID: \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\") " Oct 30 09:53:48.617490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.589843 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7e4d501-83df-4eee-9008-a1da59cd2ee1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e4d501-83df-4eee-9008-a1da59cd2ee1-plugins-config-volume\") pod \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\" (UID: \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\") " Oct 30 09:53:48.617490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.589896 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7e4d501-83df-4eee-9008-a1da59cd2ee1-kube-api-access-9c4b7\" (UniqueName: \"kubernetes.io/projected/f7e4d501-83df-4eee-9008-a1da59cd2ee1-kube-api-access-9c4b7\") pod \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\" (UID: \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\") " Oct 30 09:53:48.617490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.589949 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7e4d501-83df-4eee-9008-a1da59cd2ee1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7e4d501-83df-4eee-9008-a1da59cd2ee1-provider-credentials\") pod \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\" (UID: \"f7e4d501-83df-4eee-9008-a1da59cd2ee1\") " Oct 30 09:53:48.617490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.600370 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e4d501-83df-4eee-9008-a1da59cd2ee1-routing-overlay-volume" pod "f7e4d501-83df-4eee-9008-a1da59cd2ee1" (UID: "f7e4d501-83df-4eee-9008-a1da59cd2ee1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:48.617490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.606015 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e4d501-83df-4eee-9008-a1da59cd2ee1-kube-api-access-9c4b7" pod "f7e4d501-83df-4eee-9008-a1da59cd2ee1" (UID: "f7e4d501-83df-4eee-9008-a1da59cd2ee1"). InnerVolumeSpecName "kube-api-access-9c4b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:48.617490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.607850 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e4d501-83df-4eee-9008-a1da59cd2ee1-plugins-config-volume" pod "f7e4d501-83df-4eee-9008-a1da59cd2ee1" (UID: "f7e4d501-83df-4eee-9008-a1da59cd2ee1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:48.617490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.610102 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e4d501-83df-4eee-9008-a1da59cd2ee1-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:48.617490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.610123 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c4b7\" (UniqueName: \"kubernetes.io/projected/f7e4d501-83df-4eee-9008-a1da59cd2ee1-kube-api-access-9c4b7\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:48.617490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.610138 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7e4d501-83df-4eee-9008-a1da59cd2ee1-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:48.617490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.612539 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e4d501-83df-4eee-9008-a1da59cd2ee1-provider-credentials" pod "f7e4d501-83df-4eee-9008-a1da59cd2ee1" (UID: "f7e4d501-83df-4eee-9008-a1da59cd2ee1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:48.645611 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.620895 5938 scope.go:122] "RemoveContainer" containerID="45af0ca963909d595ab1d6d2b2c8aa45a221eb58d6ac5e5eef9d5e310f0f312b" Oct 30 09:53:48.660609 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:53:48.656759 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45af0ca963909d595ab1d6d2b2c8aa45a221eb58d6ac5e5eef9d5e310f0f312b\": container with ID starting with 45af0ca963909d595ab1d6d2b2c8aa45a221eb58d6ac5e5eef9d5e310f0f312b not found: ID does not exist" containerID="45af0ca963909d595ab1d6d2b2c8aa45a221eb58d6ac5e5eef9d5e310f0f312b" Oct 30 09:53:48.660609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.656807 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45af0ca963909d595ab1d6d2b2c8aa45a221eb58d6ac5e5eef9d5e310f0f312b"} err="failed to get container status \"45af0ca963909d595ab1d6d2b2c8aa45a221eb58d6ac5e5eef9d5e310f0f312b\": rpc error: code = NotFound desc = could not find container \"45af0ca963909d595ab1d6d2b2c8aa45a221eb58d6ac5e5eef9d5e310f0f312b\": container with ID starting with 45af0ca963909d595ab1d6d2b2c8aa45a221eb58d6ac5e5eef9d5e310f0f312b not found: ID does not exist" Oct 30 09:53:48.711446 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.711411 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7e4d501-83df-4eee-9008-a1da59cd2ee1-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:48.869658 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.865649 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz"] Oct 30 09:53:48.886642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.876903 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz"] Oct 30 09:53:48.897615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:48.896290 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-84ffnhjfz" podUID="f7e4d501-83df-4eee-9008-a1da59cd2ee1" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.215:50052\" within 1s: context deadline exceeded" Oct 30 09:53:50.056219 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:50.052488 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp"] Oct 30 09:53:50.262546 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:50.262509 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7e4d501-83df-4eee-9008-a1da59cd2ee1" path="/var/lib/kubelet/pods/f7e4d501-83df-4eee-9008-a1da59cd2ee1/volumes" Oct 30 09:53:51.313141 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:51.313110 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452"] Oct 30 09:53:52.181229 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.181196 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg"] Oct 30 09:53:52.181966 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.181950 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6102d5ea-7a81-45a0-81b0-afead85c8871" containerName="revoke-keys" Oct 30 09:53:52.181966 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.181966 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6102d5ea-7a81-45a0-81b0-afead85c8871" containerName="revoke-keys" Oct 30 09:53:52.182085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.181987 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b4dacfa7-b373-403f-8881-537132dd2997" containerName="revoke-keys" Oct 30 09:53:52.182085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.181992 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4dacfa7-b373-403f-8881-537132dd2997" containerName="revoke-keys" Oct 30 09:53:52.182085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.182019 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f7e4d501-83df-4eee-9008-a1da59cd2ee1" containerName="payload-processing" Oct 30 09:53:52.182085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.182024 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7e4d501-83df-4eee-9008-a1da59cd2ee1" containerName="payload-processing" Oct 30 09:53:52.182085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.182046 5938 cpu_manager.go:414] "containerMap: removing container" podUID="befb486a-0145-4d7e-934b-6d4d01e5aa0f" containerName="payload-processing" Oct 30 09:53:52.182085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.182051 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="befb486a-0145-4d7e-934b-6d4d01e5aa0f" containerName="payload-processing" Oct 30 09:53:52.182424 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.182150 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7e4d501-83df-4eee-9008-a1da59cd2ee1" containerName="payload-processing" Oct 30 09:53:52.182424 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.182163 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="befb486a-0145-4d7e-934b-6d4d01e5aa0f" containerName="payload-processing" Oct 30 09:53:52.182424 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.182173 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6102d5ea-7a81-45a0-81b0-afead85c8871" containerName="revoke-keys" Oct 30 09:53:52.182424 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.182179 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4dacfa7-b373-403f-8881-537132dd2997" containerName="revoke-keys" Oct 30 09:53:52.186959 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.186940 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" Oct 30 09:53:52.204191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.204169 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg"] Oct 30 09:53:52.223466 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.223426 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:53:52.223466 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.223448 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:53:52.295619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.295568 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rg6d\" (UniqueName: \"kubernetes.io/projected/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-kube-api-access-4rg6d\") pod \"maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg\" (UID: \"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" Oct 30 09:53:52.295752 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.295667 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg\" (UID: \"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" Oct 30 09:53:52.295837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.295766 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg\" (UID: \"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" Oct 30 09:53:52.396845 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.396813 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg\" (UID: \"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" Oct 30 09:53:52.397299 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.397124 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rg6d\" (UniqueName: \"kubernetes.io/projected/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-kube-api-access-4rg6d\") pod \"maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg\" (UID: \"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" Oct 30 09:53:52.397299 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.397172 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg\" (UID: \"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" Oct 30 09:53:52.398075 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.398048 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg\" (UID: \"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" Oct 30 09:53:52.399436 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.399412 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg\" (UID: \"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" Oct 30 09:53:52.417770 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.417747 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rg6d\" (UniqueName: \"kubernetes.io/projected/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-kube-api-access-4rg6d\") pod \"maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg\" (UID: \"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" Oct 30 09:53:52.502712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.502635 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" Oct 30 09:53:52.860303 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:52.860275 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg"] Oct 30 09:53:52.860452 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:53:52.860426 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cbd05d_1a1b_4b1f_b211_fa4f5dd136a7.slice/crio-7ec0c2ac9faaf5ed3ef17eed459c39dd16c3d4e7b02bab82a0bca36a1091ef86 WatchSource:0}: Error finding container 7ec0c2ac9faaf5ed3ef17eed459c39dd16c3d4e7b02bab82a0bca36a1091ef86: Status 404 returned error can't find the container with id 7ec0c2ac9faaf5ed3ef17eed459c39dd16c3d4e7b02bab82a0bca36a1091ef86 Oct 30 09:53:53.032292 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:53.032266 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht"] Oct 30 09:53:53.037533 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:53.037512 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6481d0bf-kmjht"] Oct 30 09:53:53.539100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:53.539059 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" event={"ID":"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7","Type":"ContainerStarted","Data":"942c1e2b9492029c5b4e286026b3d2efd33db2eab049ddcd9d96f0c8b808b134"} Oct 30 09:53:53.539550 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:53.539164 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" event={"ID":"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7","Type":"ContainerStarted","Data":"7ec0c2ac9faaf5ed3ef17eed459c39dd16c3d4e7b02bab82a0bca36a1091ef86"} Oct 30 09:53:53.583963 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:53.583877 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" podStartSLOduration=1.583864126 podStartE2EDuration="1.583864126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:53.579363625 +0000 UTC m=+3551.442069447" watchObservedRunningTime="2026-10-30 09:53:53.583864126 +0000 UTC m=+3551.446569947" Oct 30 09:53:53.659698 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:53.659673 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn"] Oct 30 09:53:53.693103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:53.693065 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm"] Oct 30 09:53:53.693466 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:53.693433 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerName="istio-proxy" containerID="cri-o://9a16d5c3793f14dd09569b0459516aee1c30bce9baf6d1c846946e24172dbe73" gracePeriod=30 Oct 30 09:53:54.259404 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:54.259371 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdf5c897-be06-4cf9-9bdd-33974bd190e0" path="/var/lib/kubelet/pods/bdf5c897-be06-4cf9-9bdd-33974bd190e0/volumes" Oct 30 09:53:54.549202 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:54.549117 5938 generic.go:356] "Generic (PLEG): container finished" podID="b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7" containerID="942c1e2b9492029c5b4e286026b3d2efd33db2eab049ddcd9d96f0c8b808b134" exitCode=0 Oct 30 09:53:54.549202 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:54.549188 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" event={"ID":"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7","Type":"ContainerDied","Data":"942c1e2b9492029c5b4e286026b3d2efd33db2eab049ddcd9d96f0c8b808b134"} Oct 30 09:53:55.053684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.053642 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" podUID="4b7900e2-f696-43e4-a0c9-c72bac59be25" containerName="payload-processing" containerID="cri-o://7daf74ae85f1a1baa49d96b45b30da26aaa976041f9f7f328d773a9a37dc7b94" gracePeriod=25 Oct 30 09:53:55.323012 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.322948 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:55.388227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.388194 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b7900e2-f696-43e4-a0c9-c72bac59be25-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b7900e2-f696-43e4-a0c9-c72bac59be25-provider-credentials\") pod \"4b7900e2-f696-43e4-a0c9-c72bac59be25\" (UID: \"4b7900e2-f696-43e4-a0c9-c72bac59be25\") " Oct 30 09:53:55.388440 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.388406 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b7900e2-f696-43e4-a0c9-c72bac59be25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7900e2-f696-43e4-a0c9-c72bac59be25-plugins-config-volume\") pod \"4b7900e2-f696-43e4-a0c9-c72bac59be25\" (UID: \"4b7900e2-f696-43e4-a0c9-c72bac59be25\") " Oct 30 09:53:55.388523 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.388477 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b7900e2-f696-43e4-a0c9-c72bac59be25-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b7900e2-f696-43e4-a0c9-c72bac59be25-routing-overlay-volume\") pod \"4b7900e2-f696-43e4-a0c9-c72bac59be25\" (UID: \"4b7900e2-f696-43e4-a0c9-c72bac59be25\") " Oct 30 09:53:55.388651 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.388527 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b7900e2-f696-43e4-a0c9-c72bac59be25-kube-api-access-mwqx5\" (UniqueName: \"kubernetes.io/projected/4b7900e2-f696-43e4-a0c9-c72bac59be25-kube-api-access-mwqx5\") pod \"4b7900e2-f696-43e4-a0c9-c72bac59be25\" (UID: \"4b7900e2-f696-43e4-a0c9-c72bac59be25\") " Oct 30 09:53:55.388877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.388851 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7900e2-f696-43e4-a0c9-c72bac59be25-plugins-config-volume" pod "4b7900e2-f696-43e4-a0c9-c72bac59be25" (UID: "4b7900e2-f696-43e4-a0c9-c72bac59be25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:55.389356 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.389302 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7900e2-f696-43e4-a0c9-c72bac59be25-routing-overlay-volume" pod "4b7900e2-f696-43e4-a0c9-c72bac59be25" (UID: "4b7900e2-f696-43e4-a0c9-c72bac59be25"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:55.389826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.389779 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7900e2-f696-43e4-a0c9-c72bac59be25-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.389826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.389803 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b7900e2-f696-43e4-a0c9-c72bac59be25-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.391190 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.390978 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7900e2-f696-43e4-a0c9-c72bac59be25-provider-credentials" pod "4b7900e2-f696-43e4-a0c9-c72bac59be25" (UID: "4b7900e2-f696-43e4-a0c9-c72bac59be25"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:55.392566 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.392541 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7900e2-f696-43e4-a0c9-c72bac59be25-kube-api-access-mwqx5" pod "4b7900e2-f696-43e4-a0c9-c72bac59be25" (UID: "4b7900e2-f696-43e4-a0c9-c72bac59be25"). InnerVolumeSpecName "kube-api-access-mwqx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:55.491017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.490989 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwqx5\" (UniqueName: \"kubernetes.io/projected/4b7900e2-f696-43e4-a0c9-c72bac59be25-kube-api-access-mwqx5\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.491017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.491016 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b7900e2-f696-43e4-a0c9-c72bac59be25-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.559136 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.559106 5938 generic.go:356] "Generic (PLEG): container finished" podID="4b7900e2-f696-43e4-a0c9-c72bac59be25" containerID="7daf74ae85f1a1baa49d96b45b30da26aaa976041f9f7f328d773a9a37dc7b94" exitCode=0 Oct 30 09:53:55.559509 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.559157 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" Oct 30 09:53:55.559509 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.559193 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" event={"ID":"4b7900e2-f696-43e4-a0c9-c72bac59be25","Type":"ContainerDied","Data":"7daf74ae85f1a1baa49d96b45b30da26aaa976041f9f7f328d773a9a37dc7b94"} Oct 30 09:53:55.559509 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.559257 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp" event={"ID":"4b7900e2-f696-43e4-a0c9-c72bac59be25","Type":"ContainerDied","Data":"e0a60080e5f7bb98b23520e5d6dbd54e90353d56b5b6bd67bf76b53b5fd66cd8"} Oct 30 09:53:55.559509 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.559282 5938 scope.go:122] "RemoveContainer" containerID="7daf74ae85f1a1baa49d96b45b30da26aaa976041f9f7f328d773a9a37dc7b94" Oct 30 09:53:55.571083 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.571063 5938 scope.go:122] "RemoveContainer" containerID="7daf74ae85f1a1baa49d96b45b30da26aaa976041f9f7f328d773a9a37dc7b94" Oct 30 09:53:55.571362 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:53:55.571334 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7daf74ae85f1a1baa49d96b45b30da26aaa976041f9f7f328d773a9a37dc7b94\": container with ID starting with 7daf74ae85f1a1baa49d96b45b30da26aaa976041f9f7f328d773a9a37dc7b94 not found: ID does not exist" containerID="7daf74ae85f1a1baa49d96b45b30da26aaa976041f9f7f328d773a9a37dc7b94" Oct 30 09:53:55.571464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.571374 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7daf74ae85f1a1baa49d96b45b30da26aaa976041f9f7f328d773a9a37dc7b94"} err="failed to get container status \"7daf74ae85f1a1baa49d96b45b30da26aaa976041f9f7f328d773a9a37dc7b94\": rpc error: code = NotFound desc = could not find container \"7daf74ae85f1a1baa49d96b45b30da26aaa976041f9f7f328d773a9a37dc7b94\": container with ID starting with 7daf74ae85f1a1baa49d96b45b30da26aaa976041f9f7f328d773a9a37dc7b94 not found: ID does not exist" Oct 30 09:53:55.617289 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.617258 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp"] Oct 30 09:53:55.621739 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.621710 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-546bm5zzp"] Oct 30 09:53:55.712707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.712675 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" Oct 30 09:53:55.797899 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.797863 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-maas-api-xks-ca\") pod \"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7\" (UID: \"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7\") " Oct 30 09:53:55.798117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.797962 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-kube-api-access-4rg6d\" (UniqueName: \"kubernetes.io/projected/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-kube-api-access-4rg6d\") pod \"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7\" (UID: \"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7\") " Oct 30 09:53:55.798117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.798060 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-maas-api-openshift-ca\") pod \"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7\" (UID: \"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7\") " Oct 30 09:53:55.799958 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.799922 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-maas-api-openshift-ca" pod "b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7" (UID: "b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:55.802362 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.802334 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-maas-api-xks-ca" pod "b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7" (UID: "b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:53:55.804796 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.804755 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-kube-api-access-4rg6d" pod "b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7" (UID: "b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7"). InnerVolumeSpecName "kube-api-access-4rg6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:55.900503 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.900427 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.900503 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.900455 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rg6d\" (UniqueName: \"kubernetes.io/projected/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-kube-api-access-4rg6d\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.900503 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:55.900465 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:56.279142 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.278962 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b7900e2-f696-43e4-a0c9-c72bac59be25" path="/var/lib/kubelet/pods/4b7900e2-f696-43e4-a0c9-c72bac59be25/volumes" Oct 30 09:53:56.314222 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.314193 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" podUID="645092e4-7212-451f-bbcd-70fe8ff21fa9" containerName="payload-processing" containerID="cri-o://5ff290fded5deb6a8869a97419120e73df68f4004ba86ca7036414d85d39ab86" gracePeriod=25 Oct 30 09:53:56.570476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.570439 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" Oct 30 09:53:56.570476 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.570462 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg" event={"ID":"b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7","Type":"ContainerDied","Data":"7ec0c2ac9faaf5ed3ef17eed459c39dd16c3d4e7b02bab82a0bca36a1091ef86"} Oct 30 09:53:56.570972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.570519 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ec0c2ac9faaf5ed3ef17eed459c39dd16c3d4e7b02bab82a0bca36a1091ef86" Oct 30 09:53:56.573303 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.573282 5938 generic.go:356] "Generic (PLEG): container finished" podID="645092e4-7212-451f-bbcd-70fe8ff21fa9" containerID="5ff290fded5deb6a8869a97419120e73df68f4004ba86ca7036414d85d39ab86" exitCode=0 Oct 30 09:53:56.573421 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.573357 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" event={"ID":"645092e4-7212-451f-bbcd-70fe8ff21fa9","Type":"ContainerDied","Data":"5ff290fded5deb6a8869a97419120e73df68f4004ba86ca7036414d85d39ab86"} Oct 30 09:53:56.600868 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.600845 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:56.713727 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.713699 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/645092e4-7212-451f-bbcd-70fe8ff21fa9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/645092e4-7212-451f-bbcd-70fe8ff21fa9-routing-overlay-volume\") pod \"645092e4-7212-451f-bbcd-70fe8ff21fa9\" (UID: \"645092e4-7212-451f-bbcd-70fe8ff21fa9\") " Oct 30 09:53:56.713913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.713823 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645092e4-7212-451f-bbcd-70fe8ff21fa9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/645092e4-7212-451f-bbcd-70fe8ff21fa9-provider-credentials\") pod \"645092e4-7212-451f-bbcd-70fe8ff21fa9\" (UID: \"645092e4-7212-451f-bbcd-70fe8ff21fa9\") " Oct 30 09:53:56.713913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.713862 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/645092e4-7212-451f-bbcd-70fe8ff21fa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645092e4-7212-451f-bbcd-70fe8ff21fa9-plugins-config-volume\") pod \"645092e4-7212-451f-bbcd-70fe8ff21fa9\" (UID: \"645092e4-7212-451f-bbcd-70fe8ff21fa9\") " Oct 30 09:53:56.713999 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.713932 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645092e4-7212-451f-bbcd-70fe8ff21fa9-kube-api-access-76rg6\" (UniqueName: \"kubernetes.io/projected/645092e4-7212-451f-bbcd-70fe8ff21fa9-kube-api-access-76rg6\") pod \"645092e4-7212-451f-bbcd-70fe8ff21fa9\" (UID: \"645092e4-7212-451f-bbcd-70fe8ff21fa9\") " Oct 30 09:53:56.714096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.714072 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645092e4-7212-451f-bbcd-70fe8ff21fa9-routing-overlay-volume" pod "645092e4-7212-451f-bbcd-70fe8ff21fa9" (UID: "645092e4-7212-451f-bbcd-70fe8ff21fa9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:56.714324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.714299 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645092e4-7212-451f-bbcd-70fe8ff21fa9-plugins-config-volume" pod "645092e4-7212-451f-bbcd-70fe8ff21fa9" (UID: "645092e4-7212-451f-bbcd-70fe8ff21fa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:56.714459 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.714343 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/645092e4-7212-451f-bbcd-70fe8ff21fa9-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:56.716263 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.716242 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645092e4-7212-451f-bbcd-70fe8ff21fa9-kube-api-access-76rg6" pod "645092e4-7212-451f-bbcd-70fe8ff21fa9" (UID: "645092e4-7212-451f-bbcd-70fe8ff21fa9"). InnerVolumeSpecName "kube-api-access-76rg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:56.716328 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.716266 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645092e4-7212-451f-bbcd-70fe8ff21fa9-provider-credentials" pod "645092e4-7212-451f-bbcd-70fe8ff21fa9" (UID: "645092e4-7212-451f-bbcd-70fe8ff21fa9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:56.815380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.815351 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76rg6\" (UniqueName: \"kubernetes.io/projected/645092e4-7212-451f-bbcd-70fe8ff21fa9-kube-api-access-76rg6\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:56.815380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.815380 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/645092e4-7212-451f-bbcd-70fe8ff21fa9-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:56.815530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:56.815394 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645092e4-7212-451f-bbcd-70fe8ff21fa9-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:57.244969 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.244929 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q"] Oct 30 09:53:57.245792 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.245770 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4b7900e2-f696-43e4-a0c9-c72bac59be25" containerName="payload-processing" Oct 30 09:53:57.245792 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.245792 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b7900e2-f696-43e4-a0c9-c72bac59be25" containerName="payload-processing" Oct 30 09:53:57.245991 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.245837 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7" containerName="revoke-keys" Oct 30 09:53:57.245991 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.245845 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7" containerName="revoke-keys" Oct 30 09:53:57.245991 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.245860 5938 cpu_manager.go:414] "containerMap: removing container" podUID="645092e4-7212-451f-bbcd-70fe8ff21fa9" containerName="payload-processing" Oct 30 09:53:57.245991 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.245867 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645092e4-7212-451f-bbcd-70fe8ff21fa9" containerName="payload-processing" Oct 30 09:53:57.246203 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.246096 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b7900e2-f696-43e4-a0c9-c72bac59be25" containerName="payload-processing" Oct 30 09:53:57.246203 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.246118 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7" containerName="revoke-keys" Oct 30 09:53:57.246203 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.246132 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="645092e4-7212-451f-bbcd-70fe8ff21fa9" containerName="payload-processing" Oct 30 09:53:57.250212 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.250188 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" Oct 30 09:53:57.266663 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.266638 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q"] Oct 30 09:53:57.275474 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.275449 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:53:57.275621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.275485 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:53:57.322247 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.322217 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4tzj\" (UniqueName: \"kubernetes.io/projected/66ea8d6e-5dfa-401f-ad11-46d0007dff64-kube-api-access-s4tzj\") pod \"maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q\" (UID: \"66ea8d6e-5dfa-401f-ad11-46d0007dff64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" Oct 30 09:53:57.322386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.322366 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ea8d6e-5dfa-401f-ad11-46d0007dff64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q\" (UID: \"66ea8d6e-5dfa-401f-ad11-46d0007dff64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" Oct 30 09:53:57.322623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.322572 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ea8d6e-5dfa-401f-ad11-46d0007dff64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q\" (UID: \"66ea8d6e-5dfa-401f-ad11-46d0007dff64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" Oct 30 09:53:57.424103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.424063 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4tzj\" (UniqueName: \"kubernetes.io/projected/66ea8d6e-5dfa-401f-ad11-46d0007dff64-kube-api-access-s4tzj\") pod \"maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q\" (UID: \"66ea8d6e-5dfa-401f-ad11-46d0007dff64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" Oct 30 09:53:57.424278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.424178 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ea8d6e-5dfa-401f-ad11-46d0007dff64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q\" (UID: \"66ea8d6e-5dfa-401f-ad11-46d0007dff64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" Oct 30 09:53:57.424330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.424309 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ea8d6e-5dfa-401f-ad11-46d0007dff64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q\" (UID: \"66ea8d6e-5dfa-401f-ad11-46d0007dff64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" Oct 30 09:53:57.425059 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.425030 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ea8d6e-5dfa-401f-ad11-46d0007dff64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q\" (UID: \"66ea8d6e-5dfa-401f-ad11-46d0007dff64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" Oct 30 09:53:57.426870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.426842 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ea8d6e-5dfa-401f-ad11-46d0007dff64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q\" (UID: \"66ea8d6e-5dfa-401f-ad11-46d0007dff64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" Oct 30 09:53:57.444313 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.444281 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4tzj\" (UniqueName: \"kubernetes.io/projected/66ea8d6e-5dfa-401f-ad11-46d0007dff64-kube-api-access-s4tzj\") pod \"maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q\" (UID: \"66ea8d6e-5dfa-401f-ad11-46d0007dff64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" Oct 30 09:53:57.565575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.565500 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" Oct 30 09:53:57.583357 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.583335 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" Oct 30 09:53:57.583763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.583374 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452" event={"ID":"645092e4-7212-451f-bbcd-70fe8ff21fa9","Type":"ContainerDied","Data":"8663ff8bdda33892aad461691862821a5543ceea6678dde30180a150ebb14eba"} Oct 30 09:53:57.583763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.583448 5938 scope.go:122] "RemoveContainer" containerID="5ff290fded5deb6a8869a97419120e73df68f4004ba86ca7036414d85d39ab86" Oct 30 09:53:57.655167 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.655141 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452"] Oct 30 09:53:57.661377 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.661344 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb57p452"] Oct 30 09:53:57.876475 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:57.876447 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q"] Oct 30 09:53:57.876475 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:53:57.876457 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ea8d6e_5dfa_401f_ad11_46d0007dff64.slice/crio-522043892e717c7cb91b4ae6f1fa44756c4656b2cbf00993baa024481b628f6d WatchSource:0}: Error finding container 522043892e717c7cb91b4ae6f1fa44756c4656b2cbf00993baa024481b628f6d: Status 404 returned error can't find the container with id 522043892e717c7cb91b4ae6f1fa44756c4656b2cbf00993baa024481b628f6d Oct 30 09:53:58.259580 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:58.259548 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645092e4-7212-451f-bbcd-70fe8ff21fa9" path="/var/lib/kubelet/pods/645092e4-7212-451f-bbcd-70fe8ff21fa9/volumes" Oct 30 09:53:58.594042 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:58.593948 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" event={"ID":"66ea8d6e-5dfa-401f-ad11-46d0007dff64","Type":"ContainerStarted","Data":"e0e6335b8ef181b09fd7f0814f16f5af559164e601c4412957f54b915f41c308"} Oct 30 09:53:58.594505 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:58.594046 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" event={"ID":"66ea8d6e-5dfa-401f-ad11-46d0007dff64","Type":"ContainerStarted","Data":"522043892e717c7cb91b4ae6f1fa44756c4656b2cbf00993baa024481b628f6d"} Oct 30 09:53:58.636653 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:58.636604 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" podStartSLOduration=1.636570586 podStartE2EDuration="1.636570586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:58.633493171 +0000 UTC m=+3556.496198994" watchObservedRunningTime="2026-10-30 09:53:58.636570586 +0000 UTC m=+3556.499276470" Oct 30 09:53:58.660376 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:58.660338 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" podUID="d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1" containerName="payload-processing" containerID="cri-o://aa89e993c654532f1ce6fb4cdf6bfe2e8607bbef102b03d68f2d5c6a4ddfa9b9" gracePeriod=25 Oct 30 09:53:59.111077 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.111050 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:59.114833 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.114809 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:53:59.254069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.253998 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-workload-socket\") pod \"56c07cf9-ff35-40b1-96f3-e0188c19e626\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " Oct 30 09:53:59.254225 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254091 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-workload-certs\") pod \"56c07cf9-ff35-40b1-96f3-e0188c19e626\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " Oct 30 09:53:59.254225 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254118 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-plugins-config-volume\") pod \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\" (UID: \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\") " Oct 30 09:53:59.254225 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254153 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-podinfo\") pod \"56c07cf9-ff35-40b1-96f3-e0188c19e626\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " Oct 30 09:53:59.254225 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254194 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-data\") pod \"56c07cf9-ff35-40b1-96f3-e0188c19e626\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " Oct 30 09:53:59.254414 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254242 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-envoy\") pod \"56c07cf9-ff35-40b1-96f3-e0188c19e626\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " Oct 30 09:53:59.254414 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254260 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-workload-socket" pod "56c07cf9-ff35-40b1-96f3-e0188c19e626" (UID: "56c07cf9-ff35-40b1-96f3-e0188c19e626"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:53:59.254414 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254298 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-token\" (UniqueName: \"kubernetes.io/projected/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-token\") pod \"56c07cf9-ff35-40b1-96f3-e0188c19e626\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " Oct 30 09:53:59.254414 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254364 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c07cf9-ff35-40b1-96f3-e0188c19e626-kube-api-access-hvtr6\" (UniqueName: \"kubernetes.io/projected/56c07cf9-ff35-40b1-96f3-e0188c19e626-kube-api-access-hvtr6\") pod \"56c07cf9-ff35-40b1-96f3-e0188c19e626\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " Oct 30 09:53:59.254414 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254412 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56c07cf9-ff35-40b1-96f3-e0188c19e626-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c07cf9-ff35-40b1-96f3-e0188c19e626-istiod-ca-cert\") pod \"56c07cf9-ff35-40b1-96f3-e0188c19e626\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " Oct 30 09:53:59.254714 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254419 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-workload-certs" pod "56c07cf9-ff35-40b1-96f3-e0188c19e626" (UID: "56c07cf9-ff35-40b1-96f3-e0188c19e626"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:53:59.254714 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254448 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-provider-credentials\") pod \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\" (UID: \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\") " Oct 30 09:53:59.254714 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254475 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-data" pod "56c07cf9-ff35-40b1-96f3-e0188c19e626" (UID: "56c07cf9-ff35-40b1-96f3-e0188c19e626"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:53:59.254714 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254483 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-kube-api-access-wnmjb\" (UniqueName: \"kubernetes.io/projected/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-kube-api-access-wnmjb\") pod \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\" (UID: \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\") " Oct 30 09:53:59.254714 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254553 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-credential-socket\") pod \"56c07cf9-ff35-40b1-96f3-e0188c19e626\" (UID: \"56c07cf9-ff35-40b1-96f3-e0188c19e626\") " Oct 30 09:53:59.254714 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254605 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-plugins-config-volume" pod "d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1" (UID: "d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:59.254714 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254629 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-routing-overlay-volume\") pod \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\" (UID: \"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1\") " Oct 30 09:53:59.255099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.254854 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c07cf9-ff35-40b1-96f3-e0188c19e626-istiod-ca-cert" pod "56c07cf9-ff35-40b1-96f3-e0188c19e626" (UID: "56c07cf9-ff35-40b1-96f3-e0188c19e626"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:59.255214 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.255106 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-credential-socket" pod "56c07cf9-ff35-40b1-96f3-e0188c19e626" (UID: "56c07cf9-ff35-40b1-96f3-e0188c19e626"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:53:59.255342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.255318 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-routing-overlay-volume" pod "d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1" (UID: "d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:59.255525 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.255501 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c07cf9-ff35-40b1-96f3-e0188c19e626-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:59.255614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.255533 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:59.255614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.255549 5938 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-routing-overlay-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:59.255614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.255565 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:59.255614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.255605 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:59.255783 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.255621 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:59.255783 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.255634 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:59.257076 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.257045 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-kube-api-access-wnmjb" pod "d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1" (UID: "d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1"). InnerVolumeSpecName "kube-api-access-wnmjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:59.257234 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.257199 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-provider-credentials" pod "d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1" (UID: "d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:59.257361 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.257345 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-podinfo" pod "56c07cf9-ff35-40b1-96f3-e0188c19e626" (UID: "56c07cf9-ff35-40b1-96f3-e0188c19e626"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:53:59.257462 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.257444 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-envoy" pod "56c07cf9-ff35-40b1-96f3-e0188c19e626" (UID: "56c07cf9-ff35-40b1-96f3-e0188c19e626"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:53:59.257550 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.257533 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-token" pod "56c07cf9-ff35-40b1-96f3-e0188c19e626" (UID: "56c07cf9-ff35-40b1-96f3-e0188c19e626"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:59.258204 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.258185 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c07cf9-ff35-40b1-96f3-e0188c19e626-kube-api-access-hvtr6" pod "56c07cf9-ff35-40b1-96f3-e0188c19e626" (UID: "56c07cf9-ff35-40b1-96f3-e0188c19e626"). InnerVolumeSpecName "kube-api-access-hvtr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:59.362746 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.362707 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:59.362746 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.362745 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:59.363010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.362761 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c07cf9-ff35-40b1-96f3-e0188c19e626-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:59.363010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.362776 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvtr6\" (UniqueName: \"kubernetes.io/projected/56c07cf9-ff35-40b1-96f3-e0188c19e626-kube-api-access-hvtr6\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:59.363010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.362791 5938 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-provider-credentials\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:59.363010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.362804 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnmjb\" (UniqueName: \"kubernetes.io/projected/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1-kube-api-access-wnmjb\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:53:59.402120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.402079 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf"] Oct 30 09:53:59.406632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.403489 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1" containerName="payload-processing" Oct 30 09:53:59.406632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.403517 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1" containerName="payload-processing" Oct 30 09:53:59.406632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.403540 5938 cpu_manager.go:414] "containerMap: removing container" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerName="istio-proxy" Oct 30 09:53:59.406632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.403547 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerName="istio-proxy" Oct 30 09:53:59.406632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.403759 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1" containerName="payload-processing" Oct 30 09:53:59.406632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.403782 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerName="istio-proxy" Oct 30 09:53:59.410166 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.410131 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" Oct 30 09:53:59.420610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.420568 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf"] Oct 30 09:53:59.580026 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.579941 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v929\" (UniqueName: \"kubernetes.io/projected/df5778f4-3437-4aaa-bc11-8b57a47bf552-kube-api-access-4v929\") pod \"maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf\" (UID: \"df5778f4-3437-4aaa-bc11-8b57a47bf552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" Oct 30 09:53:59.580192 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.580077 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5778f4-3437-4aaa-bc11-8b57a47bf552-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf\" (UID: \"df5778f4-3437-4aaa-bc11-8b57a47bf552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" Oct 30 09:53:59.580259 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.580245 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5778f4-3437-4aaa-bc11-8b57a47bf552-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf\" (UID: \"df5778f4-3437-4aaa-bc11-8b57a47bf552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" Oct 30 09:53:59.607005 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.606971 5938 generic.go:356] "Generic (PLEG): container finished" podID="56c07cf9-ff35-40b1-96f3-e0188c19e626" containerID="9a16d5c3793f14dd09569b0459516aee1c30bce9baf6d1c846946e24172dbe73" exitCode=0 Oct 30 09:53:59.607471 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.607083 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" Oct 30 09:53:59.607471 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.607111 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" event={"ID":"56c07cf9-ff35-40b1-96f3-e0188c19e626","Type":"ContainerDied","Data":"9a16d5c3793f14dd09569b0459516aee1c30bce9baf6d1c846946e24172dbe73"} Oct 30 09:53:59.607471 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.607171 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm" event={"ID":"56c07cf9-ff35-40b1-96f3-e0188c19e626","Type":"ContainerDied","Data":"de63f31450b480e7e5c834240383305b3860f0f037e70a45e60ad6a35b218e69"} Oct 30 09:53:59.607471 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.607201 5938 scope.go:122] "RemoveContainer" containerID="9a16d5c3793f14dd09569b0459516aee1c30bce9baf6d1c846946e24172dbe73" Oct 30 09:53:59.609369 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.609346 5938 generic.go:356] "Generic (PLEG): container finished" podID="d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1" containerID="aa89e993c654532f1ce6fb4cdf6bfe2e8607bbef102b03d68f2d5c6a4ddfa9b9" exitCode=0 Oct 30 09:53:59.609496 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.609449 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" event={"ID":"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1","Type":"ContainerDied","Data":"aa89e993c654532f1ce6fb4cdf6bfe2e8607bbef102b03d68f2d5c6a4ddfa9b9"} Oct 30 09:53:59.609582 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.609495 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" event={"ID":"d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1","Type":"ContainerDied","Data":"2aaa5d35a3c32a84fbce520fbe10e14881cff9b6113a5fb557a2f317967888d8"} Oct 30 09:53:59.609582 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.609565 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn" Oct 30 09:53:59.612867 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.612843 5938 generic.go:356] "Generic (PLEG): container finished" podID="66ea8d6e-5dfa-401f-ad11-46d0007dff64" containerID="e0e6335b8ef181b09fd7f0814f16f5af559164e601c4412957f54b915f41c308" exitCode=0 Oct 30 09:53:59.613138 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.612868 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" event={"ID":"66ea8d6e-5dfa-401f-ad11-46d0007dff64","Type":"ContainerDied","Data":"e0e6335b8ef181b09fd7f0814f16f5af559164e601c4412957f54b915f41c308"} Oct 30 09:53:59.621110 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.621074 5938 scope.go:122] "RemoveContainer" containerID="9a16d5c3793f14dd09569b0459516aee1c30bce9baf6d1c846946e24172dbe73" Oct 30 09:53:59.621519 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:53:59.621482 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a16d5c3793f14dd09569b0459516aee1c30bce9baf6d1c846946e24172dbe73\": container with ID starting with 9a16d5c3793f14dd09569b0459516aee1c30bce9baf6d1c846946e24172dbe73 not found: ID does not exist" containerID="9a16d5c3793f14dd09569b0459516aee1c30bce9baf6d1c846946e24172dbe73" Oct 30 09:53:59.621668 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.621528 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a16d5c3793f14dd09569b0459516aee1c30bce9baf6d1c846946e24172dbe73"} err="failed to get container status \"9a16d5c3793f14dd09569b0459516aee1c30bce9baf6d1c846946e24172dbe73\": rpc error: code = NotFound desc = could not find container \"9a16d5c3793f14dd09569b0459516aee1c30bce9baf6d1c846946e24172dbe73\": container with ID starting with 9a16d5c3793f14dd09569b0459516aee1c30bce9baf6d1c846946e24172dbe73 not found: ID does not exist" Oct 30 09:53:59.621668 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.621552 5938 scope.go:122] "RemoveContainer" containerID="aa89e993c654532f1ce6fb4cdf6bfe2e8607bbef102b03d68f2d5c6a4ddfa9b9" Oct 30 09:53:59.632647 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.632628 5938 scope.go:122] "RemoveContainer" containerID="aa89e993c654532f1ce6fb4cdf6bfe2e8607bbef102b03d68f2d5c6a4ddfa9b9" Oct 30 09:53:59.632947 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:53:59.632929 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa89e993c654532f1ce6fb4cdf6bfe2e8607bbef102b03d68f2d5c6a4ddfa9b9\": container with ID starting with aa89e993c654532f1ce6fb4cdf6bfe2e8607bbef102b03d68f2d5c6a4ddfa9b9 not found: ID does not exist" containerID="aa89e993c654532f1ce6fb4cdf6bfe2e8607bbef102b03d68f2d5c6a4ddfa9b9" Oct 30 09:53:59.633028 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.632957 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa89e993c654532f1ce6fb4cdf6bfe2e8607bbef102b03d68f2d5c6a4ddfa9b9"} err="failed to get container status \"aa89e993c654532f1ce6fb4cdf6bfe2e8607bbef102b03d68f2d5c6a4ddfa9b9\": rpc error: code = NotFound desc = could not find container \"aa89e993c654532f1ce6fb4cdf6bfe2e8607bbef102b03d68f2d5c6a4ddfa9b9\": container with ID starting with aa89e993c654532f1ce6fb4cdf6bfe2e8607bbef102b03d68f2d5c6a4ddfa9b9 not found: ID does not exist" Oct 30 09:53:59.668027 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.668006 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm"] Oct 30 09:53:59.672209 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.672189 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-w22tm"] Oct 30 09:53:59.681246 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.681223 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v929\" (UniqueName: \"kubernetes.io/projected/df5778f4-3437-4aaa-bc11-8b57a47bf552-kube-api-access-4v929\") pod \"maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf\" (UID: \"df5778f4-3437-4aaa-bc11-8b57a47bf552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" Oct 30 09:53:59.681401 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.681383 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5778f4-3437-4aaa-bc11-8b57a47bf552-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf\" (UID: \"df5778f4-3437-4aaa-bc11-8b57a47bf552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" Oct 30 09:53:59.681519 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.681504 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5778f4-3437-4aaa-bc11-8b57a47bf552-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf\" (UID: \"df5778f4-3437-4aaa-bc11-8b57a47bf552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" Oct 30 09:53:59.682020 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.681999 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5778f4-3437-4aaa-bc11-8b57a47bf552-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf\" (UID: \"df5778f4-3437-4aaa-bc11-8b57a47bf552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" Oct 30 09:53:59.684011 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.683994 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5778f4-3437-4aaa-bc11-8b57a47bf552-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf\" (UID: \"df5778f4-3437-4aaa-bc11-8b57a47bf552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" Oct 30 09:53:59.702400 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.702376 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v929\" (UniqueName: \"kubernetes.io/projected/df5778f4-3437-4aaa-bc11-8b57a47bf552-kube-api-access-4v929\") pod \"maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf\" (UID: \"df5778f4-3437-4aaa-bc11-8b57a47bf552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" Oct 30 09:53:59.706429 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.706407 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn"] Oct 30 09:53:59.710288 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.710268 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d66d22/payload-processing-external-model-e2e-extmod-b-d66d22-6d6czqstn"] Oct 30 09:53:59.745946 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:53:59.745115 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" Oct 30 09:54:00.062938 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:54:00.062908 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5778f4_3437_4aaa_bc11_8b57a47bf552.slice/crio-b5db62d4c5c1c66c8d84c4eb3b0bb0c8fc2c32ad32387b860f5fe6311e8de5f3 WatchSource:0}: Error finding container b5db62d4c5c1c66c8d84c4eb3b0bb0c8fc2c32ad32387b860f5fe6311e8de5f3: Status 404 returned error can't find the container with id b5db62d4c5c1c66c8d84c4eb3b0bb0c8fc2c32ad32387b860f5fe6311e8de5f3 Oct 30 09:54:00.063293 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:00.063274 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf"] Oct 30 09:54:00.259421 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:00.259380 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c07cf9-ff35-40b1-96f3-e0188c19e626" path="/var/lib/kubelet/pods/56c07cf9-ff35-40b1-96f3-e0188c19e626/volumes" Oct 30 09:54:00.259984 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:00.259962 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1" path="/var/lib/kubelet/pods/d1aa1ff0-c8cf-4289-b9cd-f792be68a9c1/volumes" Oct 30 09:54:00.624793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:00.624760 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" event={"ID":"df5778f4-3437-4aaa-bc11-8b57a47bf552","Type":"ContainerStarted","Data":"36b1b9add788e5cc7528151351f064ec8f6141b31c6e511b8c0673e9a52947bf"} Oct 30 09:54:00.625262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:00.624812 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" event={"ID":"df5778f4-3437-4aaa-bc11-8b57a47bf552","Type":"ContainerStarted","Data":"b5db62d4c5c1c66c8d84c4eb3b0bb0c8fc2c32ad32387b860f5fe6311e8de5f3"} Oct 30 09:54:00.668289 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:00.668240 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" podStartSLOduration=1.668225294 podStartE2EDuration="1.668225294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:54:00.664066264 +0000 UTC m=+3558.526772086" watchObservedRunningTime="2026-10-30 09:54:00.668225294 +0000 UTC m=+3558.530931503" Oct 30 09:54:00.794386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:00.794362 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" Oct 30 09:54:00.903058 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:00.902981 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ea8d6e-5dfa-401f-ad11-46d0007dff64-kube-api-access-s4tzj\" (UniqueName: \"kubernetes.io/projected/66ea8d6e-5dfa-401f-ad11-46d0007dff64-kube-api-access-s4tzj\") pod \"66ea8d6e-5dfa-401f-ad11-46d0007dff64\" (UID: \"66ea8d6e-5dfa-401f-ad11-46d0007dff64\") " Oct 30 09:54:00.903222 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:00.903110 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ea8d6e-5dfa-401f-ad11-46d0007dff64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ea8d6e-5dfa-401f-ad11-46d0007dff64-maas-api-xks-ca\") pod \"66ea8d6e-5dfa-401f-ad11-46d0007dff64\" (UID: \"66ea8d6e-5dfa-401f-ad11-46d0007dff64\") " Oct 30 09:54:00.903302 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:00.903233 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ea8d6e-5dfa-401f-ad11-46d0007dff64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ea8d6e-5dfa-401f-ad11-46d0007dff64-maas-api-openshift-ca\") pod \"66ea8d6e-5dfa-401f-ad11-46d0007dff64\" (UID: \"66ea8d6e-5dfa-401f-ad11-46d0007dff64\") " Oct 30 09:54:00.903634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:00.903570 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ea8d6e-5dfa-401f-ad11-46d0007dff64-maas-api-openshift-ca" pod "66ea8d6e-5dfa-401f-ad11-46d0007dff64" (UID: "66ea8d6e-5dfa-401f-ad11-46d0007dff64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:54:00.903847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:00.903832 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ea8d6e-5dfa-401f-ad11-46d0007dff64-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:00.905300 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:00.905280 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ea8d6e-5dfa-401f-ad11-46d0007dff64-maas-api-xks-ca" pod "66ea8d6e-5dfa-401f-ad11-46d0007dff64" (UID: "66ea8d6e-5dfa-401f-ad11-46d0007dff64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:54:00.905374 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:00.905337 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ea8d6e-5dfa-401f-ad11-46d0007dff64-kube-api-access-s4tzj" pod "66ea8d6e-5dfa-401f-ad11-46d0007dff64" (UID: "66ea8d6e-5dfa-401f-ad11-46d0007dff64"). InnerVolumeSpecName "kube-api-access-s4tzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:01.005609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:01.005564 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ea8d6e-5dfa-401f-ad11-46d0007dff64-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:01.005609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:01.005603 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4tzj\" (UniqueName: \"kubernetes.io/projected/66ea8d6e-5dfa-401f-ad11-46d0007dff64-kube-api-access-s4tzj\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:01.634932 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:01.634904 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" Oct 30 09:54:01.634932 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:01.634910 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q" event={"ID":"66ea8d6e-5dfa-401f-ad11-46d0007dff64","Type":"ContainerDied","Data":"522043892e717c7cb91b4ae6f1fa44756c4656b2cbf00993baa024481b628f6d"} Oct 30 09:54:01.635347 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:01.634970 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="522043892e717c7cb91b4ae6f1fa44756c4656b2cbf00993baa024481b628f6d" Oct 30 09:54:01.636893 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:01.636868 5938 generic.go:356] "Generic (PLEG): container finished" podID="df5778f4-3437-4aaa-bc11-8b57a47bf552" containerID="36b1b9add788e5cc7528151351f064ec8f6141b31c6e511b8c0673e9a52947bf" exitCode=0 Oct 30 09:54:01.637029 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:01.636947 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" event={"ID":"df5778f4-3437-4aaa-bc11-8b57a47bf552","Type":"ContainerDied","Data":"36b1b9add788e5cc7528151351f064ec8f6141b31c6e511b8c0673e9a52947bf"} Oct 30 09:54:02.744019 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:02.743820 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n"] Oct 30 09:54:02.784312 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:02.784276 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj"] Oct 30 09:54:02.824977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:02.824947 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" Oct 30 09:54:02.930249 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:02.930151 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df5778f4-3437-4aaa-bc11-8b57a47bf552-kube-api-access-4v929\" (UniqueName: \"kubernetes.io/projected/df5778f4-3437-4aaa-bc11-8b57a47bf552-kube-api-access-4v929\") pod \"df5778f4-3437-4aaa-bc11-8b57a47bf552\" (UID: \"df5778f4-3437-4aaa-bc11-8b57a47bf552\") " Oct 30 09:54:02.930249 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:02.930191 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df5778f4-3437-4aaa-bc11-8b57a47bf552-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5778f4-3437-4aaa-bc11-8b57a47bf552-maas-api-openshift-ca\") pod \"df5778f4-3437-4aaa-bc11-8b57a47bf552\" (UID: \"df5778f4-3437-4aaa-bc11-8b57a47bf552\") " Oct 30 09:54:02.930516 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:02.930361 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df5778f4-3437-4aaa-bc11-8b57a47bf552-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5778f4-3437-4aaa-bc11-8b57a47bf552-maas-api-xks-ca\") pod \"df5778f4-3437-4aaa-bc11-8b57a47bf552\" (UID: \"df5778f4-3437-4aaa-bc11-8b57a47bf552\") " Oct 30 09:54:02.930655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:02.930632 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5778f4-3437-4aaa-bc11-8b57a47bf552-maas-api-openshift-ca" pod "df5778f4-3437-4aaa-bc11-8b57a47bf552" (UID: "df5778f4-3437-4aaa-bc11-8b57a47bf552"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:54:02.931243 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:02.931217 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5778f4-3437-4aaa-bc11-8b57a47bf552-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:02.932970 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:02.932944 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5778f4-3437-4aaa-bc11-8b57a47bf552-maas-api-xks-ca" pod "df5778f4-3437-4aaa-bc11-8b57a47bf552" (UID: "df5778f4-3437-4aaa-bc11-8b57a47bf552"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:54:02.933088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:02.932975 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5778f4-3437-4aaa-bc11-8b57a47bf552-kube-api-access-4v929" pod "df5778f4-3437-4aaa-bc11-8b57a47bf552" (UID: "df5778f4-3437-4aaa-bc11-8b57a47bf552"). InnerVolumeSpecName "kube-api-access-4v929". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:03.033452 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:03.033417 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v929\" (UniqueName: \"kubernetes.io/projected/df5778f4-3437-4aaa-bc11-8b57a47bf552-kube-api-access-4v929\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:03.033452 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:03.033453 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5778f4-3437-4aaa-bc11-8b57a47bf552-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:03.654257 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:03.654222 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" Oct 30 09:54:03.654501 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:03.654257 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf" event={"ID":"df5778f4-3437-4aaa-bc11-8b57a47bf552","Type":"ContainerDied","Data":"b5db62d4c5c1c66c8d84c4eb3b0bb0c8fc2c32ad32387b860f5fe6311e8de5f3"} Oct 30 09:54:03.654501 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:03.654341 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5db62d4c5c1c66c8d84c4eb3b0bb0c8fc2c32ad32387b860f5fe6311e8de5f3" Oct 30 09:54:07.744817 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:07.744775 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" podUID="2d2f96b2-b18a-4ede-8444-312790bde84c" containerName="payload-processing" containerID="cri-o://f1807b01ddebf15c90f7a113fe11a9ba0d80e111baf12be12977a77b311d36fd" gracePeriod=25 Oct 30 09:54:07.785375 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:07.785333 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" podUID="1aa4b129-c86e-4749-9a1a-bcda6f840722" containerName="payload-pre-processing" containerID="cri-o://d89781bfb28454598d383fde8cce45a2a20758c7ec3f00e5d87b06d5dc2f42eb" gracePeriod=25 Oct 30 09:54:09.851355 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:09.851323 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx"] Oct 30 09:54:09.886404 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:09.886373 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t"] Oct 30 09:54:14.852506 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:14.852440 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" podUID="f99edfef-7092-41b9-9d12-dd378b3fa482" containerName="payload-processing" containerID="cri-o://0389144b2fd8df0861ff8a798b7dad05059a1fdb7b121cb9d81a0bcc6fcba67d" gracePeriod=25 Oct 30 09:54:14.886789 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:14.886750 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" podUID="2283ff76-dc88-4578-a61a-5dc1be3887e3" containerName="payload-pre-processing" containerID="cri-o://b47ac3506e92f9f71e375437f9756fe09117edb89df0e66bf1b8d4182f326bc6" gracePeriod=25 Oct 30 09:54:15.809747 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:54:15.809714 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:54:16.384866 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:16.384831 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" podUID="f99edfef-7092-41b9-9d12-dd378b3fa482" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:54:16.384866 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:16.384848 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" podUID="2283ff76-dc88-4578-a61a-5dc1be3887e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:54:16.875834 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:16.875793 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" podUID="1aa4b129-c86e-4749-9a1a-bcda6f840722" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:54:16.876002 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:16.875854 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" podUID="2d2f96b2-b18a-4ede-8444-312790bde84c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:54:26.384697 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:26.384656 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" podUID="2283ff76-dc88-4578-a61a-5dc1be3887e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:54:26.385207 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:26.384657 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" podUID="f99edfef-7092-41b9-9d12-dd378b3fa482" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:54:26.875295 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:26.875258 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" podUID="1aa4b129-c86e-4749-9a1a-bcda6f840722" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:54:26.875494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:26.875288 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" podUID="2d2f96b2-b18a-4ede-8444-312790bde84c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:54:28.069135 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.069107 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" Oct 30 09:54:28.101917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.101893 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" Oct 30 09:54:28.191348 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.191314 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2f96b2-b18a-4ede-8444-312790bde84c-kube-api-access-772mq\" (UniqueName: \"kubernetes.io/projected/2d2f96b2-b18a-4ede-8444-312790bde84c-kube-api-access-772mq\") pod \"2d2f96b2-b18a-4ede-8444-312790bde84c\" (UID: \"2d2f96b2-b18a-4ede-8444-312790bde84c\") " Oct 30 09:54:28.191348 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.191353 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2f96b2-b18a-4ede-8444-312790bde84c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2f96b2-b18a-4ede-8444-312790bde84c-plugins-config-volume\") pod \"2d2f96b2-b18a-4ede-8444-312790bde84c\" (UID: \"2d2f96b2-b18a-4ede-8444-312790bde84c\") " Oct 30 09:54:28.191648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.191568 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa4b129-c86e-4749-9a1a-bcda6f840722-kube-api-access-pcxfw\" (UniqueName: \"kubernetes.io/projected/1aa4b129-c86e-4749-9a1a-bcda6f840722-kube-api-access-pcxfw\") pod \"1aa4b129-c86e-4749-9a1a-bcda6f840722\" (UID: \"1aa4b129-c86e-4749-9a1a-bcda6f840722\") " Oct 30 09:54:28.191787 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.191759 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa4b129-c86e-4749-9a1a-bcda6f840722-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa4b129-c86e-4749-9a1a-bcda6f840722-plugins-config-volume\") pod \"1aa4b129-c86e-4749-9a1a-bcda6f840722\" (UID: \"1aa4b129-c86e-4749-9a1a-bcda6f840722\") " Oct 30 09:54:28.192009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.191831 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2f96b2-b18a-4ede-8444-312790bde84c-plugins-config-volume" pod "2d2f96b2-b18a-4ede-8444-312790bde84c" (UID: "2d2f96b2-b18a-4ede-8444-312790bde84c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:54:28.192209 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.192129 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa4b129-c86e-4749-9a1a-bcda6f840722-plugins-config-volume" pod "1aa4b129-c86e-4749-9a1a-bcda6f840722" (UID: "1aa4b129-c86e-4749-9a1a-bcda6f840722"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:54:28.192902 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.192876 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa4b129-c86e-4749-9a1a-bcda6f840722-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:28.193014 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.192907 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2f96b2-b18a-4ede-8444-312790bde84c-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:28.193910 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.193884 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2f96b2-b18a-4ede-8444-312790bde84c-kube-api-access-772mq" pod "2d2f96b2-b18a-4ede-8444-312790bde84c" (UID: "2d2f96b2-b18a-4ede-8444-312790bde84c"). InnerVolumeSpecName "kube-api-access-772mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:28.193990 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.193930 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa4b129-c86e-4749-9a1a-bcda6f840722-kube-api-access-pcxfw" pod "1aa4b129-c86e-4749-9a1a-bcda6f840722" (UID: "1aa4b129-c86e-4749-9a1a-bcda6f840722"). InnerVolumeSpecName "kube-api-access-pcxfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:28.294227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.294172 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-772mq\" (UniqueName: \"kubernetes.io/projected/2d2f96b2-b18a-4ede-8444-312790bde84c-kube-api-access-772mq\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:28.294227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.294198 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcxfw\" (UniqueName: \"kubernetes.io/projected/1aa4b129-c86e-4749-9a1a-bcda6f840722-kube-api-access-pcxfw\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:28.881611 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.881556 5938 generic.go:356] "Generic (PLEG): container finished" podID="2d2f96b2-b18a-4ede-8444-312790bde84c" containerID="f1807b01ddebf15c90f7a113fe11a9ba0d80e111baf12be12977a77b311d36fd" exitCode=0 Oct 30 09:54:28.882065 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.882038 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" event={"ID":"2d2f96b2-b18a-4ede-8444-312790bde84c","Type":"ContainerDied","Data":"f1807b01ddebf15c90f7a113fe11a9ba0d80e111baf12be12977a77b311d36fd"} Oct 30 09:54:28.882239 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.882223 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" event={"ID":"2d2f96b2-b18a-4ede-8444-312790bde84c","Type":"ContainerDied","Data":"bd082e42bdf371698cfeb32ad130464f08dff5bb7143b593c6d4c8d497f48485"} Oct 30 09:54:28.882365 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.882352 5938 scope.go:122] "RemoveContainer" containerID="f1807b01ddebf15c90f7a113fe11a9ba0d80e111baf12be12977a77b311d36fd" Oct 30 09:54:28.882807 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.882785 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n" Oct 30 09:54:28.893788 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.893762 5938 generic.go:356] "Generic (PLEG): container finished" podID="1aa4b129-c86e-4749-9a1a-bcda6f840722" containerID="d89781bfb28454598d383fde8cce45a2a20758c7ec3f00e5d87b06d5dc2f42eb" exitCode=0 Oct 30 09:54:28.901642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.900725 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" Oct 30 09:54:28.906377 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.906306 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" event={"ID":"1aa4b129-c86e-4749-9a1a-bcda6f840722","Type":"ContainerDied","Data":"d89781bfb28454598d383fde8cce45a2a20758c7ec3f00e5d87b06d5dc2f42eb"} Oct 30 09:54:28.906664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.906644 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj" event={"ID":"1aa4b129-c86e-4749-9a1a-bcda6f840722","Type":"ContainerDied","Data":"675745d3fa65a640b392e6956ce4f5ee16f35145c5011e704b3ccd83059d1e06"} Oct 30 09:54:28.959707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.956268 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n"] Oct 30 09:54:28.969984 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.969958 5938 scope.go:122] "RemoveContainer" containerID="f1807b01ddebf15c90f7a113fe11a9ba0d80e111baf12be12977a77b311d36fd" Oct 30 09:54:28.972427 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:54:28.971088 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1807b01ddebf15c90f7a113fe11a9ba0d80e111baf12be12977a77b311d36fd\": container with ID starting with f1807b01ddebf15c90f7a113fe11a9ba0d80e111baf12be12977a77b311d36fd not found: ID does not exist" containerID="f1807b01ddebf15c90f7a113fe11a9ba0d80e111baf12be12977a77b311d36fd" Oct 30 09:54:28.972427 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.971151 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1807b01ddebf15c90f7a113fe11a9ba0d80e111baf12be12977a77b311d36fd"} err="failed to get container status \"f1807b01ddebf15c90f7a113fe11a9ba0d80e111baf12be12977a77b311d36fd\": rpc error: code = NotFound desc = could not find container \"f1807b01ddebf15c90f7a113fe11a9ba0d80e111baf12be12977a77b311d36fd\": container with ID starting with f1807b01ddebf15c90f7a113fe11a9ba0d80e111baf12be12977a77b311d36fd not found: ID does not exist" Oct 30 09:54:28.972427 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.971180 5938 scope.go:122] "RemoveContainer" containerID="d89781bfb28454598d383fde8cce45a2a20758c7ec3f00e5d87b06d5dc2f42eb" Oct 30 09:54:28.974898 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.974805 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-035d7d-7d5b89f7b-xqn8n"] Oct 30 09:54:28.989286 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.989265 5938 scope.go:122] "RemoveContainer" containerID="d89781bfb28454598d383fde8cce45a2a20758c7ec3f00e5d87b06d5dc2f42eb" Oct 30 09:54:28.989637 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:54:28.989617 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89781bfb28454598d383fde8cce45a2a20758c7ec3f00e5d87b06d5dc2f42eb\": container with ID starting with d89781bfb28454598d383fde8cce45a2a20758c7ec3f00e5d87b06d5dc2f42eb not found: ID does not exist" containerID="d89781bfb28454598d383fde8cce45a2a20758c7ec3f00e5d87b06d5dc2f42eb" Oct 30 09:54:28.989741 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.989647 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89781bfb28454598d383fde8cce45a2a20758c7ec3f00e5d87b06d5dc2f42eb"} err="failed to get container status \"d89781bfb28454598d383fde8cce45a2a20758c7ec3f00e5d87b06d5dc2f42eb\": rpc error: code = NotFound desc = could not find container \"d89781bfb28454598d383fde8cce45a2a20758c7ec3f00e5d87b06d5dc2f42eb\": container with ID starting with d89781bfb28454598d383fde8cce45a2a20758c7ec3f00e5d87b06d5dc2f42eb not found: ID does not exist" Oct 30 09:54:29.000239 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:28.999791 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj"] Oct 30 09:54:29.003889 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:29.003870 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-035d7d-7dbd466b55-hpvrj"] Oct 30 09:54:30.261547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:30.261511 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa4b129-c86e-4749-9a1a-bcda6f840722" path="/var/lib/kubelet/pods/1aa4b129-c86e-4749-9a1a-bcda6f840722/volumes" Oct 30 09:54:30.262857 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:30.262818 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2f96b2-b18a-4ede-8444-312790bde84c" path="/var/lib/kubelet/pods/2d2f96b2-b18a-4ede-8444-312790bde84c/volumes" Oct 30 09:54:34.992478 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:34.992446 5938 generic.go:356] "Generic (PLEG): container finished" podID="f99edfef-7092-41b9-9d12-dd378b3fa482" containerID="0389144b2fd8df0861ff8a798b7dad05059a1fdb7b121cb9d81a0bcc6fcba67d" exitCode=0 Oct 30 09:54:34.992902 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:34.992519 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" event={"ID":"f99edfef-7092-41b9-9d12-dd378b3fa482","Type":"ContainerDied","Data":"0389144b2fd8df0861ff8a798b7dad05059a1fdb7b121cb9d81a0bcc6fcba67d"} Oct 30 09:54:35.187378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:35.187353 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" Oct 30 09:54:35.227478 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:35.227459 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" Oct 30 09:54:35.325178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:35.325152 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2283ff76-dc88-4578-a61a-5dc1be3887e3-kube-api-access-fx7lg\" (UniqueName: \"kubernetes.io/projected/2283ff76-dc88-4578-a61a-5dc1be3887e3-kube-api-access-fx7lg\") pod \"2283ff76-dc88-4578-a61a-5dc1be3887e3\" (UID: \"2283ff76-dc88-4578-a61a-5dc1be3887e3\") " Oct 30 09:54:35.325380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:35.325250 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99edfef-7092-41b9-9d12-dd378b3fa482-kube-api-access-zflsm\" (UniqueName: \"kubernetes.io/projected/f99edfef-7092-41b9-9d12-dd378b3fa482-kube-api-access-zflsm\") pod \"f99edfef-7092-41b9-9d12-dd378b3fa482\" (UID: \"f99edfef-7092-41b9-9d12-dd378b3fa482\") " Oct 30 09:54:35.325380 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:35.325359 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f99edfef-7092-41b9-9d12-dd378b3fa482-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f99edfef-7092-41b9-9d12-dd378b3fa482-plugins-config-volume\") pod \"f99edfef-7092-41b9-9d12-dd378b3fa482\" (UID: \"f99edfef-7092-41b9-9d12-dd378b3fa482\") " Oct 30 09:54:35.325523 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:35.325400 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2283ff76-dc88-4578-a61a-5dc1be3887e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2283ff76-dc88-4578-a61a-5dc1be3887e3-plugins-config-volume\") pod \"2283ff76-dc88-4578-a61a-5dc1be3887e3\" (UID: \"2283ff76-dc88-4578-a61a-5dc1be3887e3\") " Oct 30 09:54:35.327156 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:35.325822 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99edfef-7092-41b9-9d12-dd378b3fa482-plugins-config-volume" pod "f99edfef-7092-41b9-9d12-dd378b3fa482" (UID: "f99edfef-7092-41b9-9d12-dd378b3fa482"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:54:35.327156 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:35.325887 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2283ff76-dc88-4578-a61a-5dc1be3887e3-plugins-config-volume" pod "2283ff76-dc88-4578-a61a-5dc1be3887e3" (UID: "2283ff76-dc88-4578-a61a-5dc1be3887e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:54:35.327156 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:35.327120 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f99edfef-7092-41b9-9d12-dd378b3fa482-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:35.327156 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:35.327146 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2283ff76-dc88-4578-a61a-5dc1be3887e3-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:35.328222 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:35.328191 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99edfef-7092-41b9-9d12-dd378b3fa482-kube-api-access-zflsm" pod "f99edfef-7092-41b9-9d12-dd378b3fa482" (UID: "f99edfef-7092-41b9-9d12-dd378b3fa482"). InnerVolumeSpecName "kube-api-access-zflsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:35.328368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:35.328345 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2283ff76-dc88-4578-a61a-5dc1be3887e3-kube-api-access-fx7lg" pod "2283ff76-dc88-4578-a61a-5dc1be3887e3" (UID: "2283ff76-dc88-4578-a61a-5dc1be3887e3"). InnerVolumeSpecName "kube-api-access-fx7lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:35.428706 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:35.428631 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx7lg\" (UniqueName: \"kubernetes.io/projected/2283ff76-dc88-4578-a61a-5dc1be3887e3-kube-api-access-fx7lg\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:35.428706 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:35.428658 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zflsm\" (UniqueName: \"kubernetes.io/projected/f99edfef-7092-41b9-9d12-dd378b3fa482-kube-api-access-zflsm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:36.001781 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:36.001737 5938 generic.go:356] "Generic (PLEG): container finished" podID="2283ff76-dc88-4578-a61a-5dc1be3887e3" containerID="b47ac3506e92f9f71e375437f9756fe09117edb89df0e66bf1b8d4182f326bc6" exitCode=0 Oct 30 09:54:36.002365 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:36.001818 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" Oct 30 09:54:36.002365 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:36.001823 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" event={"ID":"2283ff76-dc88-4578-a61a-5dc1be3887e3","Type":"ContainerDied","Data":"b47ac3506e92f9f71e375437f9756fe09117edb89df0e66bf1b8d4182f326bc6"} Oct 30 09:54:36.002365 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:36.001922 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t" event={"ID":"2283ff76-dc88-4578-a61a-5dc1be3887e3","Type":"ContainerDied","Data":"758f17854f6b8c9a0c705703c0c63299ae2f8330f3bce8a112955c71193834c8"} Oct 30 09:54:36.002365 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:36.001966 5938 scope.go:122] "RemoveContainer" containerID="b47ac3506e92f9f71e375437f9756fe09117edb89df0e66bf1b8d4182f326bc6" Oct 30 09:54:36.004101 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:36.004080 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" Oct 30 09:54:36.004249 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:36.004101 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx" event={"ID":"f99edfef-7092-41b9-9d12-dd378b3fa482","Type":"ContainerDied","Data":"1c4d8f539dc3d7846669c2fa4784645fe269b2cd9d1df765ea0aa88d2be004b4"} Oct 30 09:54:36.015877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:36.015860 5938 scope.go:122] "RemoveContainer" containerID="b47ac3506e92f9f71e375437f9756fe09117edb89df0e66bf1b8d4182f326bc6" Oct 30 09:54:36.016140 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:54:36.016115 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47ac3506e92f9f71e375437f9756fe09117edb89df0e66bf1b8d4182f326bc6\": container with ID starting with b47ac3506e92f9f71e375437f9756fe09117edb89df0e66bf1b8d4182f326bc6 not found: ID does not exist" containerID="b47ac3506e92f9f71e375437f9756fe09117edb89df0e66bf1b8d4182f326bc6" Oct 30 09:54:36.016206 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:36.016150 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47ac3506e92f9f71e375437f9756fe09117edb89df0e66bf1b8d4182f326bc6"} err="failed to get container status \"b47ac3506e92f9f71e375437f9756fe09117edb89df0e66bf1b8d4182f326bc6\": rpc error: code = NotFound desc = could not find container \"b47ac3506e92f9f71e375437f9756fe09117edb89df0e66bf1b8d4182f326bc6\": container with ID starting with b47ac3506e92f9f71e375437f9756fe09117edb89df0e66bf1b8d4182f326bc6 not found: ID does not exist" Oct 30 09:54:36.016206 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:36.016168 5938 scope.go:122] "RemoveContainer" containerID="0389144b2fd8df0861ff8a798b7dad05059a1fdb7b121cb9d81a0bcc6fcba67d" Oct 30 09:54:36.057989 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:36.057968 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t"] Oct 30 09:54:36.071306 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:36.071285 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-88cbc7-6549889655-glf7t"] Oct 30 09:54:36.099040 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:36.099007 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx"] Oct 30 09:54:36.103478 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:36.103457 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-88cbc7-6f6b9d7d94-pgzjx"] Oct 30 09:54:36.258553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:36.258489 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2283ff76-dc88-4578-a61a-5dc1be3887e3" path="/var/lib/kubelet/pods/2283ff76-dc88-4578-a61a-5dc1be3887e3/volumes" Oct 30 09:54:36.258930 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:36.258916 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f99edfef-7092-41b9-9d12-dd378b3fa482" path="/var/lib/kubelet/pods/f99edfef-7092-41b9-9d12-dd378b3fa482/volumes" Oct 30 09:54:39.055812 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.055782 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq"] Oct 30 09:54:39.056281 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.055991 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" podUID="1261fad1-c20a-4074-b4cf-661b40ee5b33" containerName="maas-api" containerID="cri-o://091025fe8e08311613cada8240aa557618b74f34659eacb38ea880260066aa70" gracePeriod=30 Oct 30 09:54:39.513656 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.512555 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:54:39.608048 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.608005 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-metrics-certs\") pod \"1261fad1-c20a-4074-b4cf-661b40ee5b33\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " Oct 30 09:54:39.608223 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.608183 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-maas-api-tls\") pod \"1261fad1-c20a-4074-b4cf-661b40ee5b33\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " Oct 30 09:54:39.608291 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.608279 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1261fad1-c20a-4074-b4cf-661b40ee5b33-kube-api-access-5k8ft\" (UniqueName: \"kubernetes.io/projected/1261fad1-c20a-4074-b4cf-661b40ee5b33-kube-api-access-5k8ft\") pod \"1261fad1-c20a-4074-b4cf-661b40ee5b33\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " Oct 30 09:54:39.608328 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.608320 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1261fad1-c20a-4074-b4cf-661b40ee5b33-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1261fad1-c20a-4074-b4cf-661b40ee5b33-openshift-service-ca\") pod \"1261fad1-c20a-4074-b4cf-661b40ee5b33\" (UID: \"1261fad1-c20a-4074-b4cf-661b40ee5b33\") " Oct 30 09:54:39.609019 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.609001 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1261fad1-c20a-4074-b4cf-661b40ee5b33-openshift-service-ca" pod "1261fad1-c20a-4074-b4cf-661b40ee5b33" (UID: "1261fad1-c20a-4074-b4cf-661b40ee5b33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:54:39.611707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.611653 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1261fad1-c20a-4074-b4cf-661b40ee5b33-kube-api-access-5k8ft" pod "1261fad1-c20a-4074-b4cf-661b40ee5b33" (UID: "1261fad1-c20a-4074-b4cf-661b40ee5b33"). InnerVolumeSpecName "kube-api-access-5k8ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:39.611953 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.611933 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-metrics-certs" pod "1261fad1-c20a-4074-b4cf-661b40ee5b33" (UID: "1261fad1-c20a-4074-b4cf-661b40ee5b33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:54:39.612401 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.612371 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-maas-api-tls" pod "1261fad1-c20a-4074-b4cf-661b40ee5b33" (UID: "1261fad1-c20a-4074-b4cf-661b40ee5b33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:54:39.710568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.710490 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k8ft\" (UniqueName: \"kubernetes.io/projected/1261fad1-c20a-4074-b4cf-661b40ee5b33-kube-api-access-5k8ft\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:39.710568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.710519 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1261fad1-c20a-4074-b4cf-661b40ee5b33-openshift-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:39.710568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.710529 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-metrics-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:39.710568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.710538 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1261fad1-c20a-4074-b4cf-661b40ee5b33-maas-api-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:39.736129 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.736088 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg"] Oct 30 09:54:39.739934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:39.739908 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-035d7d-z9jdg"] Oct 30 09:54:40.039343 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:40.039259 5938 generic.go:356] "Generic (PLEG): container finished" podID="1261fad1-c20a-4074-b4cf-661b40ee5b33" containerID="091025fe8e08311613cada8240aa557618b74f34659eacb38ea880260066aa70" exitCode=0 Oct 30 09:54:40.039343 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:40.039309 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" Oct 30 09:54:40.039575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:40.039349 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" event={"ID":"1261fad1-c20a-4074-b4cf-661b40ee5b33","Type":"ContainerDied","Data":"091025fe8e08311613cada8240aa557618b74f34659eacb38ea880260066aa70"} Oct 30 09:54:40.039575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:40.039402 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq" event={"ID":"1261fad1-c20a-4074-b4cf-661b40ee5b33","Type":"ContainerDied","Data":"e8c71b93fbeb803cd4fd4b8ad30d1185c486bb037bc7168acb2e8c26fa0baab1"} Oct 30 09:54:40.039575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:40.039432 5938 scope.go:122] "RemoveContainer" containerID="091025fe8e08311613cada8240aa557618b74f34659eacb38ea880260066aa70" Oct 30 09:54:40.050930 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:40.050914 5938 scope.go:122] "RemoveContainer" containerID="091025fe8e08311613cada8240aa557618b74f34659eacb38ea880260066aa70" Oct 30 09:54:40.051165 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:54:40.051148 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"091025fe8e08311613cada8240aa557618b74f34659eacb38ea880260066aa70\": container with ID starting with 091025fe8e08311613cada8240aa557618b74f34659eacb38ea880260066aa70 not found: ID does not exist" containerID="091025fe8e08311613cada8240aa557618b74f34659eacb38ea880260066aa70" Oct 30 09:54:40.051221 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:40.051173 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"091025fe8e08311613cada8240aa557618b74f34659eacb38ea880260066aa70"} err="failed to get container status \"091025fe8e08311613cada8240aa557618b74f34659eacb38ea880260066aa70\": rpc error: code = NotFound desc = could not find container \"091025fe8e08311613cada8240aa557618b74f34659eacb38ea880260066aa70\": container with ID starting with 091025fe8e08311613cada8240aa557618b74f34659eacb38ea880260066aa70 not found: ID does not exist" Oct 30 09:54:40.096112 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:40.096078 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq"] Oct 30 09:54:40.109714 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:40.109693 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-035d7d-7d8989d6f9-bdlwq"] Oct 30 09:54:40.258359 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:40.258329 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1261fad1-c20a-4074-b4cf-661b40ee5b33" path="/var/lib/kubelet/pods/1261fad1-c20a-4074-b4cf-661b40ee5b33/volumes" Oct 30 09:54:40.258816 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:40.258801 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7" path="/var/lib/kubelet/pods/b8cbd05d-1a1b-4b1f-b211-fa4f5dd136a7/volumes" Oct 30 09:54:42.721165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:42.721128 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl"] Oct 30 09:54:42.721705 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:42.721379 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" podUID="47ee9234-06c5-400c-b8b5-575e5ed17516" containerName="maas-api" containerID="cri-o://54300a36659e6a3e61b628822738a2e69b34abb54ce1ceef255f95cce23cf8de" gracePeriod=30 Oct 30 09:54:43.125615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.111852 5938 generic.go:356] "Generic (PLEG): container finished" podID="47ee9234-06c5-400c-b8b5-575e5ed17516" containerID="54300a36659e6a3e61b628822738a2e69b34abb54ce1ceef255f95cce23cf8de" exitCode=0 Oct 30 09:54:43.125615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.112096 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" event={"ID":"47ee9234-06c5-400c-b8b5-575e5ed17516","Type":"ContainerDied","Data":"54300a36659e6a3e61b628822738a2e69b34abb54ce1ceef255f95cce23cf8de"} Oct 30 09:54:43.125615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.112152 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" event={"ID":"47ee9234-06c5-400c-b8b5-575e5ed17516","Type":"ContainerDied","Data":"1e133f4bdb772ca384cc4b0d3e3cfbe85f5bb794651a49fc415cf717ac385d76"} Oct 30 09:54:43.125615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.112177 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e133f4bdb772ca384cc4b0d3e3cfbe85f5bb794651a49fc415cf717ac385d76" Oct 30 09:54:43.125615 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.114024 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:54:43.152625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.152125 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47ee9234-06c5-400c-b8b5-575e5ed17516-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ee9234-06c5-400c-b8b5-575e5ed17516-maas-api-tls\") pod \"47ee9234-06c5-400c-b8b5-575e5ed17516\" (UID: \"47ee9234-06c5-400c-b8b5-575e5ed17516\") " Oct 30 09:54:43.152625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.152218 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ee9234-06c5-400c-b8b5-575e5ed17516-kube-api-access-sw8kw\" (UniqueName: \"kubernetes.io/projected/47ee9234-06c5-400c-b8b5-575e5ed17516-kube-api-access-sw8kw\") pod \"47ee9234-06c5-400c-b8b5-575e5ed17516\" (UID: \"47ee9234-06c5-400c-b8b5-575e5ed17516\") " Oct 30 09:54:43.152625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.152299 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47ee9234-06c5-400c-b8b5-575e5ed17516-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ee9234-06c5-400c-b8b5-575e5ed17516-metrics-certs\") pod \"47ee9234-06c5-400c-b8b5-575e5ed17516\" (UID: \"47ee9234-06c5-400c-b8b5-575e5ed17516\") " Oct 30 09:54:43.152625 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.152407 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ee9234-06c5-400c-b8b5-575e5ed17516-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ee9234-06c5-400c-b8b5-575e5ed17516-openshift-service-ca\") pod \"47ee9234-06c5-400c-b8b5-575e5ed17516\" (UID: \"47ee9234-06c5-400c-b8b5-575e5ed17516\") " Oct 30 09:54:43.182762 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.182189 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ee9234-06c5-400c-b8b5-575e5ed17516-kube-api-access-sw8kw" pod "47ee9234-06c5-400c-b8b5-575e5ed17516" (UID: "47ee9234-06c5-400c-b8b5-575e5ed17516"). InnerVolumeSpecName "kube-api-access-sw8kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:43.185370 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.185326 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ee9234-06c5-400c-b8b5-575e5ed17516-maas-api-tls" pod "47ee9234-06c5-400c-b8b5-575e5ed17516" (UID: "47ee9234-06c5-400c-b8b5-575e5ed17516"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:54:43.185812 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.185784 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ee9234-06c5-400c-b8b5-575e5ed17516-openshift-service-ca" pod "47ee9234-06c5-400c-b8b5-575e5ed17516" (UID: "47ee9234-06c5-400c-b8b5-575e5ed17516"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:54:43.194735 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.194452 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ee9234-06c5-400c-b8b5-575e5ed17516-metrics-certs" pod "47ee9234-06c5-400c-b8b5-575e5ed17516" (UID: "47ee9234-06c5-400c-b8b5-575e5ed17516"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:54:43.283627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.283164 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ee9234-06c5-400c-b8b5-575e5ed17516-openshift-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:43.283627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.283192 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ee9234-06c5-400c-b8b5-575e5ed17516-maas-api-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:43.283627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.283208 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw8kw\" (UniqueName: \"kubernetes.io/projected/47ee9234-06c5-400c-b8b5-575e5ed17516-kube-api-access-sw8kw\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:43.283627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.283218 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ee9234-06c5-400c-b8b5-575e5ed17516-metrics-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:43.526962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.526931 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q"] Oct 30 09:54:43.530303 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:43.530284 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d60c68-wjv9q"] Oct 30 09:54:44.120133 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:44.120104 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl" Oct 30 09:54:44.180846 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:44.180806 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl"] Oct 30 09:54:44.191330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:44.191299 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d60c68-75bf557c6b-rh9nl"] Oct 30 09:54:44.259165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:44.259135 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ee9234-06c5-400c-b8b5-575e5ed17516" path="/var/lib/kubelet/pods/47ee9234-06c5-400c-b8b5-575e5ed17516/volumes" Oct 30 09:54:44.259650 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:44.259633 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ea8d6e-5dfa-401f-ad11-46d0007dff64" path="/var/lib/kubelet/pods/66ea8d6e-5dfa-401f-ad11-46d0007dff64/volumes" Oct 30 09:54:44.394108 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:44.394041 5938 scope.go:122] "RemoveContainer" containerID="a1b8724c5a0af924c6ec358a678438cd72388bcd2d656b575f2ddb342be995e6" Oct 30 09:54:44.405631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:44.405609 5938 scope.go:122] "RemoveContainer" containerID="54300a36659e6a3e61b628822738a2e69b34abb54ce1ceef255f95cce23cf8de" Oct 30 09:54:45.809803 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:54:45.809767 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:54:47.149337 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.149302 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr"] Oct 30 09:54:47.150195 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150166 5938 cpu_manager.go:414] "containerMap: removing container" podUID="47ee9234-06c5-400c-b8b5-575e5ed17516" containerName="maas-api" Oct 30 09:54:47.150195 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150187 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ee9234-06c5-400c-b8b5-575e5ed17516" containerName="maas-api" Oct 30 09:54:47.150368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150220 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1aa4b129-c86e-4749-9a1a-bcda6f840722" containerName="payload-pre-processing" Oct 30 09:54:47.150368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150229 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa4b129-c86e-4749-9a1a-bcda6f840722" containerName="payload-pre-processing" Oct 30 09:54:47.150368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150247 5938 cpu_manager.go:414] "containerMap: removing container" podUID="df5778f4-3437-4aaa-bc11-8b57a47bf552" containerName="revoke-keys" Oct 30 09:54:47.150368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150252 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df5778f4-3437-4aaa-bc11-8b57a47bf552" containerName="revoke-keys" Oct 30 09:54:47.150368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150259 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1261fad1-c20a-4074-b4cf-661b40ee5b33" containerName="maas-api" Oct 30 09:54:47.150368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150264 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1261fad1-c20a-4074-b4cf-661b40ee5b33" containerName="maas-api" Oct 30 09:54:47.150368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150277 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f99edfef-7092-41b9-9d12-dd378b3fa482" containerName="payload-processing" Oct 30 09:54:47.150368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150286 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99edfef-7092-41b9-9d12-dd378b3fa482" containerName="payload-processing" Oct 30 09:54:47.150368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150293 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2283ff76-dc88-4578-a61a-5dc1be3887e3" containerName="payload-pre-processing" Oct 30 09:54:47.150368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150298 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2283ff76-dc88-4578-a61a-5dc1be3887e3" containerName="payload-pre-processing" Oct 30 09:54:47.150368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150321 5938 cpu_manager.go:414] "containerMap: removing container" podUID="66ea8d6e-5dfa-401f-ad11-46d0007dff64" containerName="revoke-keys" Oct 30 09:54:47.150368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150329 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ea8d6e-5dfa-401f-ad11-46d0007dff64" containerName="revoke-keys" Oct 30 09:54:47.150368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150359 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2d2f96b2-b18a-4ede-8444-312790bde84c" containerName="payload-processing" Oct 30 09:54:47.150368 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150366 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2f96b2-b18a-4ede-8444-312790bde84c" containerName="payload-processing" Oct 30 09:54:47.150879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150521 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="df5778f4-3437-4aaa-bc11-8b57a47bf552" containerName="revoke-keys" Oct 30 09:54:47.150879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150535 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1261fad1-c20a-4074-b4cf-661b40ee5b33" containerName="maas-api" Oct 30 09:54:47.150879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150547 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99edfef-7092-41b9-9d12-dd378b3fa482" containerName="payload-processing" Oct 30 09:54:47.150879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150556 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2283ff76-dc88-4578-a61a-5dc1be3887e3" containerName="payload-pre-processing" Oct 30 09:54:47.150879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150574 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ea8d6e-5dfa-401f-ad11-46d0007dff64" containerName="revoke-keys" Oct 30 09:54:47.150879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150602 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2f96b2-b18a-4ede-8444-312790bde84c" containerName="payload-processing" Oct 30 09:54:47.150879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150610 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ee9234-06c5-400c-b8b5-575e5ed17516" containerName="maas-api" Oct 30 09:54:47.150879 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.150618 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa4b129-c86e-4749-9a1a-bcda6f840722" containerName="payload-pre-processing" Oct 30 09:54:47.154258 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.154242 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" Oct 30 09:54:47.171906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.171881 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr"] Oct 30 09:54:47.180440 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.180414 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:54:47.180746 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.180729 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:54:47.239927 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.239894 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021b6e56-6488-465a-9a7b-0ea36a8a0285-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr\" (UID: \"021b6e56-6488-465a-9a7b-0ea36a8a0285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" Oct 30 09:54:47.240083 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.240024 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grvt6\" (UniqueName: \"kubernetes.io/projected/021b6e56-6488-465a-9a7b-0ea36a8a0285-kube-api-access-grvt6\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr\" (UID: \"021b6e56-6488-465a-9a7b-0ea36a8a0285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" Oct 30 09:54:47.240178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.240165 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021b6e56-6488-465a-9a7b-0ea36a8a0285-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr\" (UID: \"021b6e56-6488-465a-9a7b-0ea36a8a0285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" Oct 30 09:54:47.341193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.341156 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grvt6\" (UniqueName: \"kubernetes.io/projected/021b6e56-6488-465a-9a7b-0ea36a8a0285-kube-api-access-grvt6\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr\" (UID: \"021b6e56-6488-465a-9a7b-0ea36a8a0285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" Oct 30 09:54:47.341360 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.341309 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021b6e56-6488-465a-9a7b-0ea36a8a0285-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr\" (UID: \"021b6e56-6488-465a-9a7b-0ea36a8a0285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" Oct 30 09:54:47.341491 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.341469 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021b6e56-6488-465a-9a7b-0ea36a8a0285-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr\" (UID: \"021b6e56-6488-465a-9a7b-0ea36a8a0285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" Oct 30 09:54:47.342123 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.342098 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021b6e56-6488-465a-9a7b-0ea36a8a0285-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr\" (UID: \"021b6e56-6488-465a-9a7b-0ea36a8a0285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" Oct 30 09:54:47.344073 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.344051 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021b6e56-6488-465a-9a7b-0ea36a8a0285-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr\" (UID: \"021b6e56-6488-465a-9a7b-0ea36a8a0285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" Oct 30 09:54:47.362091 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.362068 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grvt6\" (UniqueName: \"kubernetes.io/projected/021b6e56-6488-465a-9a7b-0ea36a8a0285-kube-api-access-grvt6\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr\" (UID: \"021b6e56-6488-465a-9a7b-0ea36a8a0285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" Oct 30 09:54:47.469500 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.469425 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" Oct 30 09:54:47.776963 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:47.776933 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr"] Oct 30 09:54:47.777580 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:54:47.777556 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021b6e56_6488_465a_9a7b_0ea36a8a0285.slice/crio-49a47d3aabd67db22440ebd9473e2189eed97b6e4a6413ea0a2581856588c8a1 WatchSource:0}: Error finding container 49a47d3aabd67db22440ebd9473e2189eed97b6e4a6413ea0a2581856588c8a1: Status 404 returned error can't find the container with id 49a47d3aabd67db22440ebd9473e2189eed97b6e4a6413ea0a2581856588c8a1 Oct 30 09:54:48.157007 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:48.156965 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" event={"ID":"021b6e56-6488-465a-9a7b-0ea36a8a0285","Type":"ContainerStarted","Data":"b0531c41bc03a1695c6fcfec33132bd285c6d9c2bfd96e071d0e009395db0eca"} Oct 30 09:54:48.157541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:48.157037 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" event={"ID":"021b6e56-6488-465a-9a7b-0ea36a8a0285","Type":"ContainerStarted","Data":"49a47d3aabd67db22440ebd9473e2189eed97b6e4a6413ea0a2581856588c8a1"} Oct 30 09:54:48.217567 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:48.217502 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" podStartSLOduration=1.217486614 podStartE2EDuration="1.217486614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:54:48.215189943 +0000 UTC m=+3606.077895768" watchObservedRunningTime="2026-10-30 09:54:48.217486614 +0000 UTC m=+3606.080192434" Oct 30 09:54:48.969459 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:48.969422 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf"] Oct 30 09:54:48.982282 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:48.982259 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88cbc7-fhlkf"] Oct 30 09:54:49.168523 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:49.168486 5938 generic.go:356] "Generic (PLEG): container finished" podID="021b6e56-6488-465a-9a7b-0ea36a8a0285" containerID="b0531c41bc03a1695c6fcfec33132bd285c6d9c2bfd96e071d0e009395db0eca" exitCode=0 Oct 30 09:54:49.169055 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:49.168609 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" event={"ID":"021b6e56-6488-465a-9a7b-0ea36a8a0285","Type":"ContainerDied","Data":"b0531c41bc03a1695c6fcfec33132bd285c6d9c2bfd96e071d0e009395db0eca"} Oct 30 09:54:50.268214 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:50.268131 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df5778f4-3437-4aaa-bc11-8b57a47bf552" path="/var/lib/kubelet/pods/df5778f4-3437-4aaa-bc11-8b57a47bf552/volumes" Oct 30 09:54:50.330733 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:50.330711 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" Oct 30 09:54:50.497216 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:50.497148 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021b6e56-6488-465a-9a7b-0ea36a8a0285-kube-api-access-grvt6\" (UniqueName: \"kubernetes.io/projected/021b6e56-6488-465a-9a7b-0ea36a8a0285-kube-api-access-grvt6\") pod \"021b6e56-6488-465a-9a7b-0ea36a8a0285\" (UID: \"021b6e56-6488-465a-9a7b-0ea36a8a0285\") " Oct 30 09:54:50.497357 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:50.497234 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/021b6e56-6488-465a-9a7b-0ea36a8a0285-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021b6e56-6488-465a-9a7b-0ea36a8a0285-maas-api-openshift-ca\") pod \"021b6e56-6488-465a-9a7b-0ea36a8a0285\" (UID: \"021b6e56-6488-465a-9a7b-0ea36a8a0285\") " Oct 30 09:54:50.497501 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:50.497481 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/021b6e56-6488-465a-9a7b-0ea36a8a0285-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021b6e56-6488-465a-9a7b-0ea36a8a0285-maas-api-xks-ca\") pod \"021b6e56-6488-465a-9a7b-0ea36a8a0285\" (UID: \"021b6e56-6488-465a-9a7b-0ea36a8a0285\") " Oct 30 09:54:50.497573 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:50.497540 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021b6e56-6488-465a-9a7b-0ea36a8a0285-maas-api-openshift-ca" pod "021b6e56-6488-465a-9a7b-0ea36a8a0285" (UID: "021b6e56-6488-465a-9a7b-0ea36a8a0285"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:54:50.498038 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:50.498021 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021b6e56-6488-465a-9a7b-0ea36a8a0285-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:50.499551 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:50.499535 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021b6e56-6488-465a-9a7b-0ea36a8a0285-kube-api-access-grvt6" pod "021b6e56-6488-465a-9a7b-0ea36a8a0285" (UID: "021b6e56-6488-465a-9a7b-0ea36a8a0285"). InnerVolumeSpecName "kube-api-access-grvt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:50.499653 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:50.499633 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/021b6e56-6488-465a-9a7b-0ea36a8a0285-maas-api-xks-ca" pod "021b6e56-6488-465a-9a7b-0ea36a8a0285" (UID: "021b6e56-6488-465a-9a7b-0ea36a8a0285"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:54:50.600770 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:50.600729 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021b6e56-6488-465a-9a7b-0ea36a8a0285-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:50.600968 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:50.600780 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grvt6\" (UniqueName: \"kubernetes.io/projected/021b6e56-6488-465a-9a7b-0ea36a8a0285-kube-api-access-grvt6\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:51.184962 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.184935 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" Oct 30 09:54:51.185165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.184939 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr" event={"ID":"021b6e56-6488-465a-9a7b-0ea36a8a0285","Type":"ContainerDied","Data":"49a47d3aabd67db22440ebd9473e2189eed97b6e4a6413ea0a2581856588c8a1"} Oct 30 09:54:51.185165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.185059 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49a47d3aabd67db22440ebd9473e2189eed97b6e4a6413ea0a2581856588c8a1" Oct 30 09:54:51.198272 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.198248 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg"] Oct 30 09:54:51.198988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.198971 5938 cpu_manager.go:414] "containerMap: removing container" podUID="021b6e56-6488-465a-9a7b-0ea36a8a0285" containerName="revoke-keys" Oct 30 09:54:51.199044 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.198990 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021b6e56-6488-465a-9a7b-0ea36a8a0285" containerName="revoke-keys" Oct 30 09:54:51.199161 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.199149 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="021b6e56-6488-465a-9a7b-0ea36a8a0285" containerName="revoke-keys" Oct 30 09:54:51.203856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.203839 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg" Oct 30 09:54:51.224321 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.224302 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:54:51.224321 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.224312 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:54:51.234628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.229243 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg"] Oct 30 09:54:51.310469 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.310441 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdg95\" (UniqueName: \"kubernetes.io/projected/b81028f7-4663-4dab-91b6-f85de8bd2f84-kube-api-access-hdg95\") pod \"maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg\" (UID: \"b81028f7-4663-4dab-91b6-f85de8bd2f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg" Oct 30 09:54:51.310841 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.310492 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81028f7-4663-4dab-91b6-f85de8bd2f84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg\" (UID: \"b81028f7-4663-4dab-91b6-f85de8bd2f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg" Oct 30 09:54:51.310841 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.310612 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81028f7-4663-4dab-91b6-f85de8bd2f84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg\" (UID: \"b81028f7-4663-4dab-91b6-f85de8bd2f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg" Oct 30 09:54:51.411945 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.411910 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdg95\" (UniqueName: \"kubernetes.io/projected/b81028f7-4663-4dab-91b6-f85de8bd2f84-kube-api-access-hdg95\") pod \"maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg\" (UID: \"b81028f7-4663-4dab-91b6-f85de8bd2f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg" Oct 30 09:54:51.412124 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.411968 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81028f7-4663-4dab-91b6-f85de8bd2f84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg\" (UID: \"b81028f7-4663-4dab-91b6-f85de8bd2f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg" Oct 30 09:54:51.412216 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.412150 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81028f7-4663-4dab-91b6-f85de8bd2f84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg\" (UID: \"b81028f7-4663-4dab-91b6-f85de8bd2f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg" Oct 30 09:54:51.412823 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.412798 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81028f7-4663-4dab-91b6-f85de8bd2f84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg\" (UID: \"b81028f7-4663-4dab-91b6-f85de8bd2f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg" Oct 30 09:54:51.414507 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.414486 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81028f7-4663-4dab-91b6-f85de8bd2f84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg\" (UID: \"b81028f7-4663-4dab-91b6-f85de8bd2f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg" Oct 30 09:54:51.432630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.432570 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdg95\" (UniqueName: \"kubernetes.io/projected/b81028f7-4663-4dab-91b6-f85de8bd2f84-kube-api-access-hdg95\") pod \"maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg\" (UID: \"b81028f7-4663-4dab-91b6-f85de8bd2f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg" Oct 30 09:54:51.517897 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.517823 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg" Oct 30 09:54:51.821095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:51.821065 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg"] Oct 30 09:54:51.821623 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:54:51.821563 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81028f7_4663_4dab_91b6_f85de8bd2f84.slice/crio-c564fca786cd01a09e3fc29dbdcbe6c5c50ba3677ce4dc6eb5879ea93ac5c150 WatchSource:0}: Error finding container c564fca786cd01a09e3fc29dbdcbe6c5c50ba3677ce4dc6eb5879ea93ac5c150: Status 404 returned error can't find the container with id c564fca786cd01a09e3fc29dbdcbe6c5c50ba3677ce4dc6eb5879ea93ac5c150 Oct 30 09:54:52.193257 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:52.193223 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg" event={"ID":"b81028f7-4663-4dab-91b6-f85de8bd2f84","Type":"ContainerStarted","Data":"c564fca786cd01a09e3fc29dbdcbe6c5c50ba3677ce4dc6eb5879ea93ac5c150"} Oct 30 09:54:52.999998 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:54:52.999964 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81028f7_4663_4dab_91b6_f85de8bd2f84.slice/crio-conmon-ec5e45b831b1964f12417aca53695e4c38f0602941721e949f20a69b59a31d31.scope\": RecentStats: unable to find data in memory cache]" Oct 30 09:54:53.203037 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:53.202958 5938 generic.go:356] "Generic (PLEG): container finished" podID="b81028f7-4663-4dab-91b6-f85de8bd2f84" containerID="ec5e45b831b1964f12417aca53695e4c38f0602941721e949f20a69b59a31d31" exitCode=0 Oct 30 09:54:53.203187 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:53.203046 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg" event={"ID":"b81028f7-4663-4dab-91b6-f85de8bd2f84","Type":"ContainerDied","Data":"ec5e45b831b1964f12417aca53695e4c38f0602941721e949f20a69b59a31d31"} Oct 30 09:54:53.953566 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:53.953534 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt"] Oct 30 09:54:53.958524 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:53.958501 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:53.980987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:53.980966 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a479df\"/\"kube-root-ca.crt\"" Oct 30 09:54:53.981091 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:53.981054 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a479df\"/\"openshift-service-ca.crt\"" Oct 30 09:54:53.995105 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:53.995079 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a479df\"/\"test-model-a479df-kserve-self-signed-certs\"" Oct 30 09:54:54.001210 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.001182 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt"] Oct 30 09:54:54.006858 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.006831 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a479df\"/\"default-dockercfg-f6vnd\"" Oct 30 09:54:54.146258 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.146227 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f16458-2419-4f90-a9e4-1d2344e24509-tls-certs\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.146473 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.146274 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-home\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.146473 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.146451 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-model-cache\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.146611 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.146500 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-tmp-dir\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.146611 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.146563 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-dshm\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.146739 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.146724 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv8np\" (UniqueName: \"kubernetes.io/projected/52f16458-2419-4f90-a9e4-1d2344e24509-kube-api-access-jv8np\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.250877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.249627 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f16458-2419-4f90-a9e4-1d2344e24509-tls-certs\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.250877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.249720 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-home\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.250877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.249856 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-model-cache\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.250877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.249900 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-tmp-dir\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.250877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.249931 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-dshm\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.250877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.250060 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv8np\" (UniqueName: \"kubernetes.io/projected/52f16458-2419-4f90-a9e4-1d2344e24509-kube-api-access-jv8np\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.250877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.250323 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-home\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.250877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.250533 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-tmp-dir\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.250877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.250792 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-model-cache\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.289640 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.281856 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f16458-2419-4f90-a9e4-1d2344e24509-tls-certs\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.329609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.297850 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv8np\" (UniqueName: \"kubernetes.io/projected/52f16458-2419-4f90-a9e4-1d2344e24509-kube-api-access-jv8np\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.330129 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.330069 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-dshm\") pod \"test-model-a479df-kserve-7c98598c69-254jt\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.475338 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.475275 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg" Oct 30 09:54:54.562278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.562236 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b81028f7-4663-4dab-91b6-f85de8bd2f84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81028f7-4663-4dab-91b6-f85de8bd2f84-maas-api-openshift-ca\") pod \"b81028f7-4663-4dab-91b6-f85de8bd2f84\" (UID: \"b81028f7-4663-4dab-91b6-f85de8bd2f84\") " Oct 30 09:54:54.562514 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.562495 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b81028f7-4663-4dab-91b6-f85de8bd2f84-kube-api-access-hdg95\" (UniqueName: \"kubernetes.io/projected/b81028f7-4663-4dab-91b6-f85de8bd2f84-kube-api-access-hdg95\") pod \"b81028f7-4663-4dab-91b6-f85de8bd2f84\" (UID: \"b81028f7-4663-4dab-91b6-f85de8bd2f84\") " Oct 30 09:54:54.562742 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.562716 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b81028f7-4663-4dab-91b6-f85de8bd2f84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81028f7-4663-4dab-91b6-f85de8bd2f84-maas-api-xks-ca\") pod \"b81028f7-4663-4dab-91b6-f85de8bd2f84\" (UID: \"b81028f7-4663-4dab-91b6-f85de8bd2f84\") " Oct 30 09:54:54.566052 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.566023 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b81028f7-4663-4dab-91b6-f85de8bd2f84-maas-api-xks-ca" pod "b81028f7-4663-4dab-91b6-f85de8bd2f84" (UID: "b81028f7-4663-4dab-91b6-f85de8bd2f84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:54:54.566369 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.566349 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b81028f7-4663-4dab-91b6-f85de8bd2f84-maas-api-openshift-ca" pod "b81028f7-4663-4dab-91b6-f85de8bd2f84" (UID: "b81028f7-4663-4dab-91b6-f85de8bd2f84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:54:54.568971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.568948 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81028f7-4663-4dab-91b6-f85de8bd2f84-kube-api-access-hdg95" pod "b81028f7-4663-4dab-91b6-f85de8bd2f84" (UID: "b81028f7-4663-4dab-91b6-f85de8bd2f84"). InnerVolumeSpecName "kube-api-access-hdg95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:54.572482 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.572459 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:54.664760 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.664731 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81028f7-4663-4dab-91b6-f85de8bd2f84-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:54.664891 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.664762 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81028f7-4663-4dab-91b6-f85de8bd2f84-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:54.664891 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.664774 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdg95\" (UniqueName: \"kubernetes.io/projected/b81028f7-4663-4dab-91b6-f85de8bd2f84-kube-api-access-hdg95\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:54.880826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:54.880797 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt"] Oct 30 09:54:54.881200 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:54:54.881180 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52f16458_2419_4f90_a9e4_1d2344e24509.slice/crio-03ecdacc87aa1c8088b8a81a96feed6fbacb21972b60b971622b3ba3169ba816 WatchSource:0}: Error finding container 03ecdacc87aa1c8088b8a81a96feed6fbacb21972b60b971622b3ba3169ba816: Status 404 returned error can't find the container with id 03ecdacc87aa1c8088b8a81a96feed6fbacb21972b60b971622b3ba3169ba816 Oct 30 09:54:55.249669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:55.244411 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" event={"ID":"52f16458-2419-4f90-a9e4-1d2344e24509","Type":"ContainerStarted","Data":"4a945cd4e784d3bba15c7ff434101ff55ac87035fcf1f6dfb6b906bdecb379a3"} Oct 30 09:54:55.249669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:55.244466 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" event={"ID":"52f16458-2419-4f90-a9e4-1d2344e24509","Type":"ContainerStarted","Data":"03ecdacc87aa1c8088b8a81a96feed6fbacb21972b60b971622b3ba3169ba816"} Oct 30 09:54:55.249669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:55.249511 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg" event={"ID":"b81028f7-4663-4dab-91b6-f85de8bd2f84","Type":"ContainerDied","Data":"c564fca786cd01a09e3fc29dbdcbe6c5c50ba3677ce4dc6eb5879ea93ac5c150"} Oct 30 09:54:55.249669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:55.249551 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c564fca786cd01a09e3fc29dbdcbe6c5c50ba3677ce4dc6eb5879ea93ac5c150" Oct 30 09:54:55.250320 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:55.249888 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg" Oct 30 09:54:55.302734 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:55.302681 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" podStartSLOduration=2.302654535 podStartE2EDuration="2.302654535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:54:55.300045907 +0000 UTC m=+3613.162751728" watchObservedRunningTime="2026-10-30 09:54:55.302654535 +0000 UTC m=+3613.165360357" Oct 30 09:54:55.574748 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:55.574659 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:54:56.632546 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.632510 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676"] Oct 30 09:54:56.633227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.633212 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b81028f7-4663-4dab-91b6-f85de8bd2f84" containerName="revoke-keys" Oct 30 09:54:56.633275 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.633229 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b81028f7-4663-4dab-91b6-f85de8bd2f84" containerName="revoke-keys" Oct 30 09:54:56.633377 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.633367 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b81028f7-4663-4dab-91b6-f85de8bd2f84" containerName="revoke-keys" Oct 30 09:54:56.637268 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.637250 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676" Oct 30 09:54:56.654387 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.654355 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676"] Oct 30 09:54:56.664152 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.664123 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:54:56.664444 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.664426 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:54:56.708727 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.708695 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff49453a-1f4b-4713-9f4e-b284ad428c7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d66d22-df676\" (UID: \"ff49453a-1f4b-4713-9f4e-b284ad428c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676" Oct 30 09:54:56.709009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.708984 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff49453a-1f4b-4713-9f4e-b284ad428c7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d66d22-df676\" (UID: \"ff49453a-1f4b-4713-9f4e-b284ad428c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676" Oct 30 09:54:56.709112 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.709028 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj64r\" (UniqueName: \"kubernetes.io/projected/ff49453a-1f4b-4713-9f4e-b284ad428c7f-kube-api-access-fj64r\") pod \"maas-api-revoke-keys-e2e-extmod-b-d66d22-df676\" (UID: \"ff49453a-1f4b-4713-9f4e-b284ad428c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676" Oct 30 09:54:56.810406 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.810377 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff49453a-1f4b-4713-9f4e-b284ad428c7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d66d22-df676\" (UID: \"ff49453a-1f4b-4713-9f4e-b284ad428c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676" Oct 30 09:54:56.810644 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.810577 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff49453a-1f4b-4713-9f4e-b284ad428c7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d66d22-df676\" (UID: \"ff49453a-1f4b-4713-9f4e-b284ad428c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676" Oct 30 09:54:56.810731 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.810649 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj64r\" (UniqueName: \"kubernetes.io/projected/ff49453a-1f4b-4713-9f4e-b284ad428c7f-kube-api-access-fj64r\") pod \"maas-api-revoke-keys-e2e-extmod-b-d66d22-df676\" (UID: \"ff49453a-1f4b-4713-9f4e-b284ad428c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676" Oct 30 09:54:56.811185 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.811163 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff49453a-1f4b-4713-9f4e-b284ad428c7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d66d22-df676\" (UID: \"ff49453a-1f4b-4713-9f4e-b284ad428c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676" Oct 30 09:54:56.813238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.813215 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff49453a-1f4b-4713-9f4e-b284ad428c7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d66d22-df676\" (UID: \"ff49453a-1f4b-4713-9f4e-b284ad428c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676" Oct 30 09:54:56.831225 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.831198 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj64r\" (UniqueName: \"kubernetes.io/projected/ff49453a-1f4b-4713-9f4e-b284ad428c7f-kube-api-access-fj64r\") pod \"maas-api-revoke-keys-e2e-extmod-b-d66d22-df676\" (UID: \"ff49453a-1f4b-4713-9f4e-b284ad428c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676" Oct 30 09:54:56.952011 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:56.951953 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676" Oct 30 09:54:57.254188 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:57.254155 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676"] Oct 30 09:54:57.254383 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:54:57.254358 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff49453a_1f4b_4713_9f4e_b284ad428c7f.slice/crio-b1bed7fa6dcd0f31da293dc2cd6d1b0d0f082c22596726ebd17692b8193b718f WatchSource:0}: Error finding container b1bed7fa6dcd0f31da293dc2cd6d1b0d0f082c22596726ebd17692b8193b718f: Status 404 returned error can't find the container with id b1bed7fa6dcd0f31da293dc2cd6d1b0d0f082c22596726ebd17692b8193b718f Oct 30 09:54:57.288958 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:57.288930 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676" event={"ID":"ff49453a-1f4b-4713-9f4e-b284ad428c7f","Type":"ContainerStarted","Data":"b1bed7fa6dcd0f31da293dc2cd6d1b0d0f082c22596726ebd17692b8193b718f"} Oct 30 09:54:58.334559 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:58.334512 5938 generic.go:356] "Generic (PLEG): container finished" podID="ff49453a-1f4b-4713-9f4e-b284ad428c7f" containerID="1e8c877b99f3e980e841d177506d2711a4362c27fb8872306f9e39450c1bd9bd" exitCode=0 Oct 30 09:54:58.335489 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:58.335462 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676" event={"ID":"ff49453a-1f4b-4713-9f4e-b284ad428c7f","Type":"ContainerDied","Data":"1e8c877b99f3e980e841d177506d2711a4362c27fb8872306f9e39450c1bd9bd"} Oct 30 09:54:59.505506 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:59.505479 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676" Oct 30 09:54:59.668634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:59.668576 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff49453a-1f4b-4713-9f4e-b284ad428c7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff49453a-1f4b-4713-9f4e-b284ad428c7f-maas-api-openshift-ca\") pod \"ff49453a-1f4b-4713-9f4e-b284ad428c7f\" (UID: \"ff49453a-1f4b-4713-9f4e-b284ad428c7f\") " Oct 30 09:54:59.668799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:59.668675 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff49453a-1f4b-4713-9f4e-b284ad428c7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff49453a-1f4b-4713-9f4e-b284ad428c7f-maas-api-xks-ca\") pod \"ff49453a-1f4b-4713-9f4e-b284ad428c7f\" (UID: \"ff49453a-1f4b-4713-9f4e-b284ad428c7f\") " Oct 30 09:54:59.668799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:59.668759 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff49453a-1f4b-4713-9f4e-b284ad428c7f-kube-api-access-fj64r\" (UniqueName: \"kubernetes.io/projected/ff49453a-1f4b-4713-9f4e-b284ad428c7f-kube-api-access-fj64r\") pod \"ff49453a-1f4b-4713-9f4e-b284ad428c7f\" (UID: \"ff49453a-1f4b-4713-9f4e-b284ad428c7f\") " Oct 30 09:54:59.669003 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:59.668981 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff49453a-1f4b-4713-9f4e-b284ad428c7f-maas-api-openshift-ca" pod "ff49453a-1f4b-4713-9f4e-b284ad428c7f" (UID: "ff49453a-1f4b-4713-9f4e-b284ad428c7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:54:59.669362 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:59.669342 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff49453a-1f4b-4713-9f4e-b284ad428c7f-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:59.670980 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:59.670951 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff49453a-1f4b-4713-9f4e-b284ad428c7f-maas-api-xks-ca" pod "ff49453a-1f4b-4713-9f4e-b284ad428c7f" (UID: "ff49453a-1f4b-4713-9f4e-b284ad428c7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:54:59.671102 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:59.671068 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff49453a-1f4b-4713-9f4e-b284ad428c7f-kube-api-access-fj64r" pod "ff49453a-1f4b-4713-9f4e-b284ad428c7f" (UID: "ff49453a-1f4b-4713-9f4e-b284ad428c7f"). InnerVolumeSpecName "kube-api-access-fj64r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:59.770808 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:59.770786 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff49453a-1f4b-4713-9f4e-b284ad428c7f-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:54:59.770808 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:54:59.770809 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj64r\" (UniqueName: \"kubernetes.io/projected/ff49453a-1f4b-4713-9f4e-b284ad428c7f-kube-api-access-fj64r\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:00.351140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:00.351112 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676" Oct 30 09:55:00.351330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:00.351143 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676" event={"ID":"ff49453a-1f4b-4713-9f4e-b284ad428c7f","Type":"ContainerDied","Data":"b1bed7fa6dcd0f31da293dc2cd6d1b0d0f082c22596726ebd17692b8193b718f"} Oct 30 09:55:00.351330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:00.351215 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1bed7fa6dcd0f31da293dc2cd6d1b0d0f082c22596726ebd17692b8193b718f" Oct 30 09:55:04.573122 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:04.573077 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:55:04.586495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:04.586472 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:55:05.415142 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:05.415113 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:55:07.318643 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:07.318610 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p"] Oct 30 09:55:12.319925 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:12.319856 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" podUID="245627dd-ae75-45aa-9f14-af4797b0d87c" containerName="payload-pre-processing" containerID="cri-o://ae49e4592e70694f4a4154ab6fee1cc87c7182945a3b38811b7f41dc5aeb5fc1" gracePeriod=25 Oct 30 09:55:15.809737 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:55:15.809704 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:55:19.291204 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:19.291162 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" podUID="245627dd-ae75-45aa-9f14-af4797b0d87c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:19.951785 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:19.951747 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr"] Oct 30 09:55:19.965574 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:19.965547 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fc7ee7-d89wr"] Oct 30 09:55:20.258913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:20.258843 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="021b6e56-6488-465a-9a7b-0ea36a8a0285" path="/var/lib/kubelet/pods/021b6e56-6488-465a-9a7b-0ea36a8a0285/volumes" Oct 30 09:55:27.014852 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.014814 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn"] Oct 30 09:55:27.015520 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.015112 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" podUID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" containerName="istio-proxy" containerID="cri-o://af5c573f116b3832ccbea722bf7993e9b4f182d910b2810cb8448d1d3735de65" gracePeriod=30 Oct 30 09:55:27.611402 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.611365 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c"] Oct 30 09:55:27.612101 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.612086 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ff49453a-1f4b-4713-9f4e-b284ad428c7f" containerName="revoke-keys" Oct 30 09:55:27.612159 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.612103 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff49453a-1f4b-4713-9f4e-b284ad428c7f" containerName="revoke-keys" Oct 30 09:55:27.612225 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.612216 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff49453a-1f4b-4713-9f4e-b284ad428c7f" containerName="revoke-keys" Oct 30 09:55:27.617314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.617292 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c" Oct 30 09:55:27.634909 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.634885 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c"] Oct 30 09:55:27.644342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.644318 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:55:27.644480 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.644327 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:55:27.695425 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.695397 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66cc0994-09e1-4863-b875-9451b5b7f1b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c\" (UID: \"66cc0994-09e1-4863-b875-9451b5b7f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c" Oct 30 09:55:27.695567 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.695444 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66cc0994-09e1-4863-b875-9451b5b7f1b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c\" (UID: \"66cc0994-09e1-4863-b875-9451b5b7f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c" Oct 30 09:55:27.695567 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.695541 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqp86\" (UniqueName: \"kubernetes.io/projected/66cc0994-09e1-4863-b875-9451b5b7f1b5-kube-api-access-kqp86\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c\" (UID: \"66cc0994-09e1-4863-b875-9451b5b7f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c" Oct 30 09:55:27.796463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.796433 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66cc0994-09e1-4863-b875-9451b5b7f1b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c\" (UID: \"66cc0994-09e1-4863-b875-9451b5b7f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c" Oct 30 09:55:27.796652 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.796484 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66cc0994-09e1-4863-b875-9451b5b7f1b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c\" (UID: \"66cc0994-09e1-4863-b875-9451b5b7f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c" Oct 30 09:55:27.796652 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.796523 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqp86\" (UniqueName: \"kubernetes.io/projected/66cc0994-09e1-4863-b875-9451b5b7f1b5-kube-api-access-kqp86\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c\" (UID: \"66cc0994-09e1-4863-b875-9451b5b7f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c" Oct 30 09:55:27.797268 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.797246 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66cc0994-09e1-4863-b875-9451b5b7f1b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c\" (UID: \"66cc0994-09e1-4863-b875-9451b5b7f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c" Oct 30 09:55:27.799098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.799076 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66cc0994-09e1-4863-b875-9451b5b7f1b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c\" (UID: \"66cc0994-09e1-4863-b875-9451b5b7f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c" Oct 30 09:55:27.817170 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.817136 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqp86\" (UniqueName: \"kubernetes.io/projected/66cc0994-09e1-4863-b875-9451b5b7f1b5-kube-api-access-kqp86\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c\" (UID: \"66cc0994-09e1-4863-b875-9451b5b7f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c" Oct 30 09:55:27.933010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:27.932982 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c" Oct 30 09:55:28.309169 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:28.309139 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c"] Oct 30 09:55:28.309900 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:55:28.309874 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66cc0994_09e1_4863_b875_9451b5b7f1b5.slice/crio-c7c490add08f80be14a54fc7b426400415d0ba1ca57529572c7974f798099c41 WatchSource:0}: Error finding container c7c490add08f80be14a54fc7b426400415d0ba1ca57529572c7974f798099c41: Status 404 returned error can't find the container with id c7c490add08f80be14a54fc7b426400415d0ba1ca57529572c7974f798099c41 Oct 30 09:55:28.592986 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:28.592923 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c" event={"ID":"66cc0994-09e1-4863-b875-9451b5b7f1b5","Type":"ContainerStarted","Data":"c7c490add08f80be14a54fc7b426400415d0ba1ca57529572c7974f798099c41"} Oct 30 09:55:29.291096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:29.291057 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" podUID="245627dd-ae75-45aa-9f14-af4797b0d87c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:29.602707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:29.602579 5938 generic.go:356] "Generic (PLEG): container finished" podID="66cc0994-09e1-4863-b875-9451b5b7f1b5" containerID="0dc8ee77f71b8929cf385ca6260784eaee8872e21edfd6821f9cabfbb4de2319" exitCode=0 Oct 30 09:55:29.602707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:29.602669 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c" event={"ID":"66cc0994-09e1-4863-b875-9451b5b7f1b5","Type":"ContainerDied","Data":"0dc8ee77f71b8929cf385ca6260784eaee8872e21edfd6821f9cabfbb4de2319"} Oct 30 09:55:30.517652 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:30.517617 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:55:30.517853 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:30.517687 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" podUID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:55:30.759546 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:30.759523 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c" Oct 30 09:55:30.841961 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:30.841891 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66cc0994-09e1-4863-b875-9451b5b7f1b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66cc0994-09e1-4863-b875-9451b5b7f1b5-maas-api-openshift-ca\") pod \"66cc0994-09e1-4863-b875-9451b5b7f1b5\" (UID: \"66cc0994-09e1-4863-b875-9451b5b7f1b5\") " Oct 30 09:55:30.842140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:30.842121 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66cc0994-09e1-4863-b875-9451b5b7f1b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66cc0994-09e1-4863-b875-9451b5b7f1b5-maas-api-xks-ca\") pod \"66cc0994-09e1-4863-b875-9451b5b7f1b5\" (UID: \"66cc0994-09e1-4863-b875-9451b5b7f1b5\") " Oct 30 09:55:30.842255 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:30.842239 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66cc0994-09e1-4863-b875-9451b5b7f1b5-kube-api-access-kqp86\" (UniqueName: \"kubernetes.io/projected/66cc0994-09e1-4863-b875-9451b5b7f1b5-kube-api-access-kqp86\") pod \"66cc0994-09e1-4863-b875-9451b5b7f1b5\" (UID: \"66cc0994-09e1-4863-b875-9451b5b7f1b5\") " Oct 30 09:55:30.842350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:30.842297 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66cc0994-09e1-4863-b875-9451b5b7f1b5-maas-api-openshift-ca" pod "66cc0994-09e1-4863-b875-9451b5b7f1b5" (UID: "66cc0994-09e1-4863-b875-9451b5b7f1b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:30.842873 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:30.842848 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66cc0994-09e1-4863-b875-9451b5b7f1b5-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:30.844911 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:30.844887 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66cc0994-09e1-4863-b875-9451b5b7f1b5-maas-api-xks-ca" pod "66cc0994-09e1-4863-b875-9451b5b7f1b5" (UID: "66cc0994-09e1-4863-b875-9451b5b7f1b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:55:30.845057 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:30.845038 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66cc0994-09e1-4863-b875-9451b5b7f1b5-kube-api-access-kqp86" pod "66cc0994-09e1-4863-b875-9451b5b7f1b5" (UID: "66cc0994-09e1-4863-b875-9451b5b7f1b5"). InnerVolumeSpecName "kube-api-access-kqp86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:30.943905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:30.943881 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66cc0994-09e1-4863-b875-9451b5b7f1b5-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:30.943905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:30.943902 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqp86\" (UniqueName: \"kubernetes.io/projected/66cc0994-09e1-4863-b875-9451b5b7f1b5-kube-api-access-kqp86\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:31.619569 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:31.619534 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c" Oct 30 09:55:31.619761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:31.619574 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c" event={"ID":"66cc0994-09e1-4863-b875-9451b5b7f1b5","Type":"ContainerDied","Data":"c7c490add08f80be14a54fc7b426400415d0ba1ca57529572c7974f798099c41"} Oct 30 09:55:31.619761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:31.619654 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7c490add08f80be14a54fc7b426400415d0ba1ca57529572c7974f798099c41" Oct 30 09:55:32.307659 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.307637 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:55:32.359564 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.359538 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istiod-ca-cert\") pod \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " Oct 30 09:55:32.359706 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.359640 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-workload-socket\") pod \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " Oct 30 09:55:32.359706 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.359695 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-credential-socket\") pod \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " Oct 30 09:55:32.359848 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.359732 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-podinfo\") pod \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " Oct 30 09:55:32.359848 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.359768 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-data\") pod \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " Oct 30 09:55:32.359967 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.359907 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-kube-api-access-d79hh\" (UniqueName: \"kubernetes.io/projected/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-kube-api-access-d79hh\") pod \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " Oct 30 09:55:32.360024 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.359986 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-envoy\") pod \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " Oct 30 09:55:32.360138 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.360034 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-workload-certs\") pod \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " Oct 30 09:55:32.360138 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.359957 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-workload-socket" pod "be2573c7-dd0a-4ba6-8e74-66cd5eef4361" (UID: "be2573c7-dd0a-4ba6-8e74-66cd5eef4361"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:32.360138 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.360059 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istiod-ca-cert" pod "be2573c7-dd0a-4ba6-8e74-66cd5eef4361" (UID: "be2573c7-dd0a-4ba6-8e74-66cd5eef4361"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:32.360138 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.360077 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-token\" (UniqueName: \"kubernetes.io/projected/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-token\") pod \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\" (UID: \"be2573c7-dd0a-4ba6-8e74-66cd5eef4361\") " Oct 30 09:55:32.360138 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.360030 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-credential-socket" pod "be2573c7-dd0a-4ba6-8e74-66cd5eef4361" (UID: "be2573c7-dd0a-4ba6-8e74-66cd5eef4361"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:32.360138 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.360092 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-data" pod "be2573c7-dd0a-4ba6-8e74-66cd5eef4361" (UID: "be2573c7-dd0a-4ba6-8e74-66cd5eef4361"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:32.360605 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.360259 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-workload-certs" pod "be2573c7-dd0a-4ba6-8e74-66cd5eef4361" (UID: "be2573c7-dd0a-4ba6-8e74-66cd5eef4361"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:32.361318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.361293 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:32.361418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.361324 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:32.361418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.361339 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:32.361418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.361353 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:32.361418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.361367 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:32.362273 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.362251 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-podinfo" pod "be2573c7-dd0a-4ba6-8e74-66cd5eef4361" (UID: "be2573c7-dd0a-4ba6-8e74-66cd5eef4361"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:55:32.362632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.362606 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-envoy" pod "be2573c7-dd0a-4ba6-8e74-66cd5eef4361" (UID: "be2573c7-dd0a-4ba6-8e74-66cd5eef4361"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:32.362737 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.362723 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-kube-api-access-d79hh" pod "be2573c7-dd0a-4ba6-8e74-66cd5eef4361" (UID: "be2573c7-dd0a-4ba6-8e74-66cd5eef4361"). InnerVolumeSpecName "kube-api-access-d79hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:32.362784 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.362746 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-token" pod "be2573c7-dd0a-4ba6-8e74-66cd5eef4361" (UID: "be2573c7-dd0a-4ba6-8e74-66cd5eef4361"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:32.462925 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.462898 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d79hh\" (UniqueName: \"kubernetes.io/projected/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-kube-api-access-d79hh\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:32.462925 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.462927 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:32.463096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.462937 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:32.463096 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.462944 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2573c7-dd0a-4ba6-8e74-66cd5eef4361-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:32.571634 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.571601 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" Oct 30 09:55:32.649035 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.642922 5938 generic.go:356] "Generic (PLEG): container finished" podID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" containerID="af5c573f116b3832ccbea722bf7993e9b4f182d910b2810cb8448d1d3735de65" exitCode=0 Oct 30 09:55:32.649035 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.643089 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" event={"ID":"be2573c7-dd0a-4ba6-8e74-66cd5eef4361","Type":"ContainerDied","Data":"af5c573f116b3832ccbea722bf7993e9b4f182d910b2810cb8448d1d3735de65"} Oct 30 09:55:32.649035 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.643189 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" event={"ID":"be2573c7-dd0a-4ba6-8e74-66cd5eef4361","Type":"ContainerDied","Data":"0a1f8b5cbb669e68237801ee85414a4a98d7dff6113325f1077750c5bd63ec5d"} Oct 30 09:55:32.649035 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.643235 5938 scope.go:122] "RemoveContainer" containerID="af5c573f116b3832ccbea722bf7993e9b4f182d910b2810cb8448d1d3735de65" Oct 30 09:55:32.649035 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.643474 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn" Oct 30 09:55:32.649035 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.646061 5938 generic.go:356] "Generic (PLEG): container finished" podID="245627dd-ae75-45aa-9f14-af4797b0d87c" containerID="ae49e4592e70694f4a4154ab6fee1cc87c7182945a3b38811b7f41dc5aeb5fc1" exitCode=0 Oct 30 09:55:32.649035 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.646283 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" event={"ID":"245627dd-ae75-45aa-9f14-af4797b0d87c","Type":"ContainerDied","Data":"ae49e4592e70694f4a4154ab6fee1cc87c7182945a3b38811b7f41dc5aeb5fc1"} Oct 30 09:55:32.649035 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.646367 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" event={"ID":"245627dd-ae75-45aa-9f14-af4797b0d87c","Type":"ContainerDied","Data":"1f6963992d8751f69e174f9227f477444691480ab2ed49dc00e0b984dd6bc5a3"} Oct 30 09:55:32.649035 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.646576 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p" Oct 30 09:55:32.665103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.665072 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/245627dd-ae75-45aa-9f14-af4797b0d87c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/245627dd-ae75-45aa-9f14-af4797b0d87c-plugins-config-volume\") pod \"245627dd-ae75-45aa-9f14-af4797b0d87c\" (UID: \"245627dd-ae75-45aa-9f14-af4797b0d87c\") " Oct 30 09:55:32.665938 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.665916 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/245627dd-ae75-45aa-9f14-af4797b0d87c-kube-api-access-n255w\" (UniqueName: \"kubernetes.io/projected/245627dd-ae75-45aa-9f14-af4797b0d87c-kube-api-access-n255w\") pod \"245627dd-ae75-45aa-9f14-af4797b0d87c\" (UID: \"245627dd-ae75-45aa-9f14-af4797b0d87c\") " Oct 30 09:55:32.666500 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.665700 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245627dd-ae75-45aa-9f14-af4797b0d87c-plugins-config-volume" pod "245627dd-ae75-45aa-9f14-af4797b0d87c" (UID: "245627dd-ae75-45aa-9f14-af4797b0d87c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:32.667150 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.667125 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/245627dd-ae75-45aa-9f14-af4797b0d87c-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:32.669939 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.669913 5938 scope.go:122] "RemoveContainer" containerID="3d9a61ef06c9603c8464098f20d12f669ba8b8640bebbf1de71840e54a7aabb6" Oct 30 09:55:32.680251 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.680061 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245627dd-ae75-45aa-9f14-af4797b0d87c-kube-api-access-n255w" pod "245627dd-ae75-45aa-9f14-af4797b0d87c" (UID: "245627dd-ae75-45aa-9f14-af4797b0d87c"). InnerVolumeSpecName "kube-api-access-n255w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:32.683663 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.683639 5938 scope.go:122] "RemoveContainer" containerID="af5c573f116b3832ccbea722bf7993e9b4f182d910b2810cb8448d1d3735de65" Oct 30 09:55:32.684032 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:55:32.684003 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5c573f116b3832ccbea722bf7993e9b4f182d910b2810cb8448d1d3735de65\": container with ID starting with af5c573f116b3832ccbea722bf7993e9b4f182d910b2810cb8448d1d3735de65 not found: ID does not exist" containerID="af5c573f116b3832ccbea722bf7993e9b4f182d910b2810cb8448d1d3735de65" Oct 30 09:55:32.684120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.684043 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5c573f116b3832ccbea722bf7993e9b4f182d910b2810cb8448d1d3735de65"} err="failed to get container status \"af5c573f116b3832ccbea722bf7993e9b4f182d910b2810cb8448d1d3735de65\": rpc error: code = NotFound desc = could not find container \"af5c573f116b3832ccbea722bf7993e9b4f182d910b2810cb8448d1d3735de65\": container with ID starting with af5c573f116b3832ccbea722bf7993e9b4f182d910b2810cb8448d1d3735de65 not found: ID does not exist" Oct 30 09:55:32.684120 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.684069 5938 scope.go:122] "RemoveContainer" containerID="3d9a61ef06c9603c8464098f20d12f669ba8b8640bebbf1de71840e54a7aabb6" Oct 30 09:55:32.684440 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:55:32.684415 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d9a61ef06c9603c8464098f20d12f669ba8b8640bebbf1de71840e54a7aabb6\": container with ID starting with 3d9a61ef06c9603c8464098f20d12f669ba8b8640bebbf1de71840e54a7aabb6 not found: ID does not exist" containerID="3d9a61ef06c9603c8464098f20d12f669ba8b8640bebbf1de71840e54a7aabb6" Oct 30 09:55:32.684528 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.684448 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9a61ef06c9603c8464098f20d12f669ba8b8640bebbf1de71840e54a7aabb6"} err="failed to get container status \"3d9a61ef06c9603c8464098f20d12f669ba8b8640bebbf1de71840e54a7aabb6\": rpc error: code = NotFound desc = could not find container \"3d9a61ef06c9603c8464098f20d12f669ba8b8640bebbf1de71840e54a7aabb6\": container with ID starting with 3d9a61ef06c9603c8464098f20d12f669ba8b8640bebbf1de71840e54a7aabb6 not found: ID does not exist" Oct 30 09:55:32.684528 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.684468 5938 scope.go:122] "RemoveContainer" containerID="ae49e4592e70694f4a4154ab6fee1cc87c7182945a3b38811b7f41dc5aeb5fc1" Oct 30 09:55:32.697981 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.697965 5938 scope.go:122] "RemoveContainer" containerID="ae49e4592e70694f4a4154ab6fee1cc87c7182945a3b38811b7f41dc5aeb5fc1" Oct 30 09:55:32.698296 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:55:32.698272 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae49e4592e70694f4a4154ab6fee1cc87c7182945a3b38811b7f41dc5aeb5fc1\": container with ID starting with ae49e4592e70694f4a4154ab6fee1cc87c7182945a3b38811b7f41dc5aeb5fc1 not found: ID does not exist" containerID="ae49e4592e70694f4a4154ab6fee1cc87c7182945a3b38811b7f41dc5aeb5fc1" Oct 30 09:55:32.698403 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.698302 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae49e4592e70694f4a4154ab6fee1cc87c7182945a3b38811b7f41dc5aeb5fc1"} err="failed to get container status \"ae49e4592e70694f4a4154ab6fee1cc87c7182945a3b38811b7f41dc5aeb5fc1\": rpc error: code = NotFound desc = could not find container \"ae49e4592e70694f4a4154ab6fee1cc87c7182945a3b38811b7f41dc5aeb5fc1\": container with ID starting with ae49e4592e70694f4a4154ab6fee1cc87c7182945a3b38811b7f41dc5aeb5fc1 not found: ID does not exist" Oct 30 09:55:32.720894 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.720839 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn"] Oct 30 09:55:32.725412 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.725390 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fc7ee7-openshift-default-697d474cb7-ks4qn"] Oct 30 09:55:32.768645 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:32.768609 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n255w\" (UniqueName: \"kubernetes.io/projected/245627dd-ae75-45aa-9f14-af4797b0d87c-kube-api-access-n255w\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:33.002575 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:33.002535 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p"] Oct 30 09:55:33.008143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:33.008111 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d66d22-54ddd954bc-5vz6p"] Oct 30 09:55:33.378939 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:33.378904 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg"] Oct 30 09:55:33.382550 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:33.382521 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3d9b36-chrcg"] Oct 30 09:55:34.258107 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:34.258074 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="245627dd-ae75-45aa-9f14-af4797b0d87c" path="/var/lib/kubelet/pods/245627dd-ae75-45aa-9f14-af4797b0d87c/volumes" Oct 30 09:55:34.258607 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:34.258571 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b81028f7-4663-4dab-91b6-f85de8bd2f84" path="/var/lib/kubelet/pods/b81028f7-4663-4dab-91b6-f85de8bd2f84/volumes" Oct 30 09:55:34.259075 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:34.259059 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" path="/var/lib/kubelet/pods/be2573c7-dd0a-4ba6-8e74-66cd5eef4361/volumes" Oct 30 09:55:37.227605 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:37.227556 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676"] Oct 30 09:55:37.233927 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:37.233895 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d66d22-df676"] Oct 30 09:55:38.203983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:38.203951 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87"] Oct 30 09:55:38.243124 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:38.243091 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq"] Oct 30 09:55:38.258119 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:38.258092 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff49453a-1f4b-4713-9f4e-b284ad428c7f" path="/var/lib/kubelet/pods/ff49453a-1f4b-4713-9f4e-b284ad428c7f/volumes" Oct 30 09:55:40.449848 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:40.449815 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565"] Oct 30 09:55:40.450264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:40.450032 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" podUID="b9985d34-9195-47d7-8e4c-fb5a87ca969a" containerName="istio-proxy" containerID="cri-o://b149e7d962d7ff3102b9755392e52267b1a27cedd8da786acc266fa14840cb57" gracePeriod=30 Oct 30 09:55:41.030670 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.030636 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk"] Oct 30 09:55:41.031315 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.031301 5938 cpu_manager.go:414] "containerMap: removing container" podUID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" containerName="istio-proxy" Oct 30 09:55:41.031376 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.031316 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" containerName="istio-proxy" Oct 30 09:55:41.031376 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.031331 5938 cpu_manager.go:414] "containerMap: removing container" podUID="245627dd-ae75-45aa-9f14-af4797b0d87c" containerName="payload-pre-processing" Oct 30 09:55:41.031376 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.031336 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245627dd-ae75-45aa-9f14-af4797b0d87c" containerName="payload-pre-processing" Oct 30 09:55:41.031478 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.031380 5938 cpu_manager.go:414] "containerMap: removing container" podUID="66cc0994-09e1-4863-b875-9451b5b7f1b5" containerName="revoke-keys" Oct 30 09:55:41.031478 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.031385 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66cc0994-09e1-4863-b875-9451b5b7f1b5" containerName="revoke-keys" Oct 30 09:55:41.031544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.031482 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" containerName="istio-proxy" Oct 30 09:55:41.031544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.031492 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="245627dd-ae75-45aa-9f14-af4797b0d87c" containerName="payload-pre-processing" Oct 30 09:55:41.031544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.031506 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="66cc0994-09e1-4863-b875-9451b5b7f1b5" containerName="revoke-keys" Oct 30 09:55:41.036405 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.036385 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk" Oct 30 09:55:41.053353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.053325 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk"] Oct 30 09:55:41.061848 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.061808 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:55:41.062436 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.062412 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:55:41.108094 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.108063 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9596f991-d4dc-469a-bf22-b87b5a3f47a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk\" (UID: \"9596f991-d4dc-469a-bf22-b87b5a3f47a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk" Oct 30 09:55:41.108229 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.108104 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9596f991-d4dc-469a-bf22-b87b5a3f47a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk\" (UID: \"9596f991-d4dc-469a-bf22-b87b5a3f47a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk" Oct 30 09:55:41.108287 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.108251 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7tdn\" (UniqueName: \"kubernetes.io/projected/9596f991-d4dc-469a-bf22-b87b5a3f47a2-kube-api-access-c7tdn\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk\" (UID: \"9596f991-d4dc-469a-bf22-b87b5a3f47a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk" Oct 30 09:55:41.209406 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.209377 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7tdn\" (UniqueName: \"kubernetes.io/projected/9596f991-d4dc-469a-bf22-b87b5a3f47a2-kube-api-access-c7tdn\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk\" (UID: \"9596f991-d4dc-469a-bf22-b87b5a3f47a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk" Oct 30 09:55:41.209581 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.209521 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9596f991-d4dc-469a-bf22-b87b5a3f47a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk\" (UID: \"9596f991-d4dc-469a-bf22-b87b5a3f47a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk" Oct 30 09:55:41.209581 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.209540 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9596f991-d4dc-469a-bf22-b87b5a3f47a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk\" (UID: \"9596f991-d4dc-469a-bf22-b87b5a3f47a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk" Oct 30 09:55:41.210236 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.210216 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9596f991-d4dc-469a-bf22-b87b5a3f47a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk\" (UID: \"9596f991-d4dc-469a-bf22-b87b5a3f47a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk" Oct 30 09:55:41.212010 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.211989 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9596f991-d4dc-469a-bf22-b87b5a3f47a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk\" (UID: \"9596f991-d4dc-469a-bf22-b87b5a3f47a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk" Oct 30 09:55:41.229966 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.229944 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7tdn\" (UniqueName: \"kubernetes.io/projected/9596f991-d4dc-469a-bf22-b87b5a3f47a2-kube-api-access-c7tdn\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk\" (UID: \"9596f991-d4dc-469a-bf22-b87b5a3f47a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk" Oct 30 09:55:41.350854 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.350795 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk" Oct 30 09:55:41.657906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.657880 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk"] Oct 30 09:55:41.658908 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:55:41.658884 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9596f991_d4dc_469a_bf22_b87b5a3f47a2.slice/crio-e37186509f7720e63e9ce046bbceefbf7af73617a81e766fa11788ea8bf741f1 WatchSource:0}: Error finding container e37186509f7720e63e9ce046bbceefbf7af73617a81e766fa11788ea8bf741f1: Status 404 returned error can't find the container with id e37186509f7720e63e9ce046bbceefbf7af73617a81e766fa11788ea8bf741f1 Oct 30 09:55:41.735830 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:41.735804 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk" event={"ID":"9596f991-d4dc-469a-bf22-b87b5a3f47a2","Type":"ContainerStarted","Data":"e37186509f7720e63e9ce046bbceefbf7af73617a81e766fa11788ea8bf741f1"} Oct 30 09:55:42.745260 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:42.745224 5938 generic.go:356] "Generic (PLEG): container finished" podID="9596f991-d4dc-469a-bf22-b87b5a3f47a2" containerID="15b077cf3d86627a2bf34e632e4d0f143d3a897cc56ee83bd21539f8b917fdf1" exitCode=0 Oct 30 09:55:42.745663 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:42.745313 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk" event={"ID":"9596f991-d4dc-469a-bf22-b87b5a3f47a2","Type":"ContainerDied","Data":"15b077cf3d86627a2bf34e632e4d0f143d3a897cc56ee83bd21539f8b917fdf1"} Oct 30 09:55:43.204895 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:43.204857 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" podUID="0bd2cc71-7527-48bf-a58a-db25175e2621" containerName="payload-processing" containerID="cri-o://e752ab005a48d54d767a016ecaf3713df0512be0508bacef957b0d1578b78756" gracePeriod=25 Oct 30 09:55:43.243404 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:43.243362 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" podUID="1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24" containerName="payload-pre-processing" containerID="cri-o://ba9e7c5ed8cc7393bfdcc1ceb208605fe092c6e70553aa0b93330638f77427b3" gracePeriod=25 Oct 30 09:55:43.938530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:43.938498 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw"] Oct 30 09:55:43.939034 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:43.938868 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" podUID="df118d0b-7613-4edf-aaa4-af1944531fa6" containerName="istio-proxy" containerID="cri-o://e6304487555bea67797bf74ea39ed434367bb1dfb0daa7232cb858002603102e" gracePeriod=30 Oct 30 09:55:43.942292 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:43.942210 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk" Oct 30 09:55:44.058742 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.058659 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9596f991-d4dc-469a-bf22-b87b5a3f47a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9596f991-d4dc-469a-bf22-b87b5a3f47a2-maas-api-xks-ca\") pod \"9596f991-d4dc-469a-bf22-b87b5a3f47a2\" (UID: \"9596f991-d4dc-469a-bf22-b87b5a3f47a2\") " Oct 30 09:55:44.058900 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.058811 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9596f991-d4dc-469a-bf22-b87b5a3f47a2-kube-api-access-c7tdn\" (UniqueName: \"kubernetes.io/projected/9596f991-d4dc-469a-bf22-b87b5a3f47a2-kube-api-access-c7tdn\") pod \"9596f991-d4dc-469a-bf22-b87b5a3f47a2\" (UID: \"9596f991-d4dc-469a-bf22-b87b5a3f47a2\") " Oct 30 09:55:44.058900 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.058844 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9596f991-d4dc-469a-bf22-b87b5a3f47a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9596f991-d4dc-469a-bf22-b87b5a3f47a2-maas-api-openshift-ca\") pod \"9596f991-d4dc-469a-bf22-b87b5a3f47a2\" (UID: \"9596f991-d4dc-469a-bf22-b87b5a3f47a2\") " Oct 30 09:55:44.059302 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.059273 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9596f991-d4dc-469a-bf22-b87b5a3f47a2-maas-api-openshift-ca" pod "9596f991-d4dc-469a-bf22-b87b5a3f47a2" (UID: "9596f991-d4dc-469a-bf22-b87b5a3f47a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:44.059774 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.059736 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9596f991-d4dc-469a-bf22-b87b5a3f47a2-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:44.061427 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.061379 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9596f991-d4dc-469a-bf22-b87b5a3f47a2-kube-api-access-c7tdn" pod "9596f991-d4dc-469a-bf22-b87b5a3f47a2" (UID: "9596f991-d4dc-469a-bf22-b87b5a3f47a2"). InnerVolumeSpecName "kube-api-access-c7tdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:44.061560 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.061421 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9596f991-d4dc-469a-bf22-b87b5a3f47a2-maas-api-xks-ca" pod "9596f991-d4dc-469a-bf22-b87b5a3f47a2" (UID: "9596f991-d4dc-469a-bf22-b87b5a3f47a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:55:44.167825 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.163935 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9596f991-d4dc-469a-bf22-b87b5a3f47a2-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:44.167825 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.163976 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7tdn\" (UniqueName: \"kubernetes.io/projected/9596f991-d4dc-469a-bf22-b87b5a3f47a2-kube-api-access-c7tdn\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:44.497327 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.497291 5938 scope.go:122] "RemoveContainer" containerID="b04f9bf46449ae38e351ca350a1d7fa1c39a7291c616ad7b86a32780d2a4c104" Oct 30 09:55:44.516608 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.516550 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5"] Oct 30 09:55:44.518277 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.518244 5938 cpu_manager.go:414] "containerMap: removing container" podUID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" containerName="istio-proxy" Oct 30 09:55:44.518277 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.518268 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" containerName="istio-proxy" Oct 30 09:55:44.518493 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.518421 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9596f991-d4dc-469a-bf22-b87b5a3f47a2" containerName="revoke-keys" Oct 30 09:55:44.518493 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.518430 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9596f991-d4dc-469a-bf22-b87b5a3f47a2" containerName="revoke-keys" Oct 30 09:55:44.522530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.518761 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9596f991-d4dc-469a-bf22-b87b5a3f47a2" containerName="revoke-keys" Oct 30 09:55:44.522530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.518784 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2573c7-dd0a-4ba6-8e74-66cd5eef4361" containerName="istio-proxy" Oct 30 09:55:44.530306 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.530285 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" Oct 30 09:55:44.540184 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.540159 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5"] Oct 30 09:55:44.568088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.568059 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d409189c-887a-4f67-a7d4-acf3ef0273c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5\" (UID: \"d409189c-887a-4f67-a7d4-acf3ef0273c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" Oct 30 09:55:44.568253 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.568231 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b876x\" (UniqueName: \"kubernetes.io/projected/d409189c-887a-4f67-a7d4-acf3ef0273c8-kube-api-access-b876x\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5\" (UID: \"d409189c-887a-4f67-a7d4-acf3ef0273c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" Oct 30 09:55:44.568330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.568290 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d409189c-887a-4f67-a7d4-acf3ef0273c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5\" (UID: \"d409189c-887a-4f67-a7d4-acf3ef0273c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" Oct 30 09:55:44.669227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.669197 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d409189c-887a-4f67-a7d4-acf3ef0273c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5\" (UID: \"d409189c-887a-4f67-a7d4-acf3ef0273c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" Oct 30 09:55:44.669451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.669330 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d409189c-887a-4f67-a7d4-acf3ef0273c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5\" (UID: \"d409189c-887a-4f67-a7d4-acf3ef0273c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" Oct 30 09:55:44.669531 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.669499 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b876x\" (UniqueName: \"kubernetes.io/projected/d409189c-887a-4f67-a7d4-acf3ef0273c8-kube-api-access-b876x\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5\" (UID: \"d409189c-887a-4f67-a7d4-acf3ef0273c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" Oct 30 09:55:44.670146 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.670121 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d409189c-887a-4f67-a7d4-acf3ef0273c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5\" (UID: \"d409189c-887a-4f67-a7d4-acf3ef0273c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" Oct 30 09:55:44.671636 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.671617 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d409189c-887a-4f67-a7d4-acf3ef0273c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5\" (UID: \"d409189c-887a-4f67-a7d4-acf3ef0273c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" Oct 30 09:55:44.690260 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.690240 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b876x\" (UniqueName: \"kubernetes.io/projected/d409189c-887a-4f67-a7d4-acf3ef0273c8-kube-api-access-b876x\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5\" (UID: \"d409189c-887a-4f67-a7d4-acf3ef0273c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" Oct 30 09:55:44.761396 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.761333 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk" Oct 30 09:55:44.761396 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.761366 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk" event={"ID":"9596f991-d4dc-469a-bf22-b87b5a3f47a2","Type":"ContainerDied","Data":"e37186509f7720e63e9ce046bbceefbf7af73617a81e766fa11788ea8bf741f1"} Oct 30 09:55:44.761552 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.761415 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e37186509f7720e63e9ce046bbceefbf7af73617a81e766fa11788ea8bf741f1" Oct 30 09:55:44.865303 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:44.865279 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" Oct 30 09:55:45.086038 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.085964 5938 patch_prober.go:29] interesting pod/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:55:45.086038 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.086017 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" podUID="df118d0b-7613-4edf-aaa4-af1944531fa6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:55:45.173764 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.173737 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5"] Oct 30 09:55:45.174892 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:55:45.174865 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd409189c_887a_4f67_a7d4_acf3ef0273c8.slice/crio-9581d05937a6013fcb3893b4070a820164bcd5831e3a7448ec0df9d2c3bcd3b2 WatchSource:0}: Error finding container 9581d05937a6013fcb3893b4070a820164bcd5831e3a7448ec0df9d2c3bcd3b2: Status 404 returned error can't find the container with id 9581d05937a6013fcb3893b4070a820164bcd5831e3a7448ec0df9d2c3bcd3b2 Oct 30 09:55:45.707511 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.707488 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:55:45.776033 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.775130 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" event={"ID":"d409189c-887a-4f67-a7d4-acf3ef0273c8","Type":"ContainerStarted","Data":"a8ad664f851c58b428a67bc445ec79db6f63b5e76ffe4a59d70cc5d799d1a82b"} Oct 30 09:55:45.776033 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.775197 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" event={"ID":"d409189c-887a-4f67-a7d4-acf3ef0273c8","Type":"ContainerStarted","Data":"9581d05937a6013fcb3893b4070a820164bcd5831e3a7448ec0df9d2c3bcd3b2"} Oct 30 09:55:45.777576 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.777537 5938 generic.go:356] "Generic (PLEG): container finished" podID="b9985d34-9195-47d7-8e4c-fb5a87ca969a" containerID="b149e7d962d7ff3102b9755392e52267b1a27cedd8da786acc266fa14840cb57" exitCode=0 Oct 30 09:55:45.777747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.777621 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" event={"ID":"b9985d34-9195-47d7-8e4c-fb5a87ca969a","Type":"ContainerDied","Data":"b149e7d962d7ff3102b9755392e52267b1a27cedd8da786acc266fa14840cb57"} Oct 30 09:55:45.777747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.777658 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" Oct 30 09:55:45.777747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.777683 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565" event={"ID":"b9985d34-9195-47d7-8e4c-fb5a87ca969a","Type":"ContainerDied","Data":"18277057b4f7404055cd4bdaa58931e4a3753ee83ca290317843892b05a95c88"} Oct 30 09:55:45.777747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.777721 5938 scope.go:122] "RemoveContainer" containerID="b149e7d962d7ff3102b9755392e52267b1a27cedd8da786acc266fa14840cb57" Oct 30 09:55:45.782851 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.782399 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-podinfo\") pod \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " Oct 30 09:55:45.782851 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.782448 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-workload-socket\") pod \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " Oct 30 09:55:45.782851 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.782746 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-workload-socket" pod "b9985d34-9195-47d7-8e4c-fb5a87ca969a" (UID: "b9985d34-9195-47d7-8e4c-fb5a87ca969a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:45.783119 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.782860 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-envoy\") pod \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " Oct 30 09:55:45.783119 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.782933 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-data\") pod \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " Oct 30 09:55:45.783119 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.782999 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-workload-certs\") pod \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " Oct 30 09:55:45.783119 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.783079 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-token\" (UniqueName: \"kubernetes.io/projected/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-token\") pod \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " Oct 30 09:55:45.783119 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.783116 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istiod-ca-cert\") pod \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " Oct 30 09:55:45.783407 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.783141 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9985d34-9195-47d7-8e4c-fb5a87ca969a-kube-api-access-22nnt\" (UniqueName: \"kubernetes.io/projected/b9985d34-9195-47d7-8e4c-fb5a87ca969a-kube-api-access-22nnt\") pod \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " Oct 30 09:55:45.783407 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.783207 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-credential-socket\") pod \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\" (UID: \"b9985d34-9195-47d7-8e4c-fb5a87ca969a\") " Oct 30 09:55:45.784662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.783951 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:45.784662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.784045 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-credential-socket" pod "b9985d34-9195-47d7-8e4c-fb5a87ca969a" (UID: "b9985d34-9195-47d7-8e4c-fb5a87ca969a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:45.784662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.784627 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-data" pod "b9985d34-9195-47d7-8e4c-fb5a87ca969a" (UID: "b9985d34-9195-47d7-8e4c-fb5a87ca969a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:45.784911 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.784885 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istiod-ca-cert" pod "b9985d34-9195-47d7-8e4c-fb5a87ca969a" (UID: "b9985d34-9195-47d7-8e4c-fb5a87ca969a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:45.785772 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.785385 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-workload-certs" pod "b9985d34-9195-47d7-8e4c-fb5a87ca969a" (UID: "b9985d34-9195-47d7-8e4c-fb5a87ca969a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:45.787068 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.787006 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9985d34-9195-47d7-8e4c-fb5a87ca969a-kube-api-access-22nnt" pod "b9985d34-9195-47d7-8e4c-fb5a87ca969a" (UID: "b9985d34-9195-47d7-8e4c-fb5a87ca969a"). InnerVolumeSpecName "kube-api-access-22nnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:45.792164 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.792049 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-podinfo" pod "b9985d34-9195-47d7-8e4c-fb5a87ca969a" (UID: "b9985d34-9195-47d7-8e4c-fb5a87ca969a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:55:45.792164 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.792130 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-token" pod "b9985d34-9195-47d7-8e4c-fb5a87ca969a" (UID: "b9985d34-9195-47d7-8e4c-fb5a87ca969a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:45.793925 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.793897 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-envoy" pod "b9985d34-9195-47d7-8e4c-fb5a87ca969a" (UID: "b9985d34-9195-47d7-8e4c-fb5a87ca969a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:45.794851 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.794832 5938 scope.go:122] "RemoveContainer" containerID="062de02c198f95e5e1ae6f5449e1d2dc709521f9184312d820d57dc67dc7fbf9" Oct 30 09:55:45.808648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.808382 5938 scope.go:122] "RemoveContainer" containerID="b149e7d962d7ff3102b9755392e52267b1a27cedd8da786acc266fa14840cb57" Oct 30 09:55:45.809051 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:55:45.808894 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b149e7d962d7ff3102b9755392e52267b1a27cedd8da786acc266fa14840cb57\": container with ID starting with b149e7d962d7ff3102b9755392e52267b1a27cedd8da786acc266fa14840cb57 not found: ID does not exist" containerID="b149e7d962d7ff3102b9755392e52267b1a27cedd8da786acc266fa14840cb57" Oct 30 09:55:45.809051 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.808936 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b149e7d962d7ff3102b9755392e52267b1a27cedd8da786acc266fa14840cb57"} err="failed to get container status \"b149e7d962d7ff3102b9755392e52267b1a27cedd8da786acc266fa14840cb57\": rpc error: code = NotFound desc = could not find container \"b149e7d962d7ff3102b9755392e52267b1a27cedd8da786acc266fa14840cb57\": container with ID starting with b149e7d962d7ff3102b9755392e52267b1a27cedd8da786acc266fa14840cb57 not found: ID does not exist" Oct 30 09:55:45.809051 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.808963 5938 scope.go:122] "RemoveContainer" containerID="062de02c198f95e5e1ae6f5449e1d2dc709521f9184312d820d57dc67dc7fbf9" Oct 30 09:55:45.809304 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:55:45.809285 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"062de02c198f95e5e1ae6f5449e1d2dc709521f9184312d820d57dc67dc7fbf9\": container with ID starting with 062de02c198f95e5e1ae6f5449e1d2dc709521f9184312d820d57dc67dc7fbf9 not found: ID does not exist" containerID="062de02c198f95e5e1ae6f5449e1d2dc709521f9184312d820d57dc67dc7fbf9" Oct 30 09:55:45.809353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.809306 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"062de02c198f95e5e1ae6f5449e1d2dc709521f9184312d820d57dc67dc7fbf9"} err="failed to get container status \"062de02c198f95e5e1ae6f5449e1d2dc709521f9184312d820d57dc67dc7fbf9\": rpc error: code = NotFound desc = could not find container \"062de02c198f95e5e1ae6f5449e1d2dc709521f9184312d820d57dc67dc7fbf9\": container with ID starting with 062de02c198f95e5e1ae6f5449e1d2dc709521f9184312d820d57dc67dc7fbf9 not found: ID does not exist" Oct 30 09:55:45.817309 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:55:45.817277 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:55:45.818517 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.818474 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" podStartSLOduration=1.818462057 podStartE2EDuration="1.818462057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:55:45.815889937 +0000 UTC m=+3663.678595781" watchObservedRunningTime="2026-10-30 09:55:45.818462057 +0000 UTC m=+3663.681167874" Oct 30 09:55:45.885430 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.885405 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:45.885430 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.885429 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:45.885648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.885439 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:45.885648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.885447 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:45.885648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.885455 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:45.885648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.885462 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9985d34-9195-47d7-8e4c-fb5a87ca969a-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:45.885648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.885470 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22nnt\" (UniqueName: \"kubernetes.io/projected/b9985d34-9195-47d7-8e4c-fb5a87ca969a-kube-api-access-22nnt\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:45.885648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:45.885479 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9985d34-9195-47d7-8e4c-fb5a87ca969a-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:46.129542 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:46.129516 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565"] Oct 30 09:55:46.132951 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:46.132929 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-3d9b36-openshift-default-7d4d98b88f-rv565"] Oct 30 09:55:46.259088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:46.259058 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9985d34-9195-47d7-8e4c-fb5a87ca969a" path="/var/lib/kubelet/pods/b9985d34-9195-47d7-8e4c-fb5a87ca969a/volumes" Oct 30 09:55:46.787284 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:46.787256 5938 generic.go:356] "Generic (PLEG): container finished" podID="d409189c-887a-4f67-a7d4-acf3ef0273c8" containerID="a8ad664f851c58b428a67bc445ec79db6f63b5e76ffe4a59d70cc5d799d1a82b" exitCode=0 Oct 30 09:55:46.787462 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:46.787323 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" event={"ID":"d409189c-887a-4f67-a7d4-acf3ef0273c8","Type":"ContainerDied","Data":"a8ad664f851c58b428a67bc445ec79db6f63b5e76ffe4a59d70cc5d799d1a82b"} Oct 30 09:55:47.087640 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:47.087520 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" podUID="1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:47.087640 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:47.087559 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" podUID="0bd2cc71-7527-48bf-a58a-db25175e2621" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:47.959945 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:47.959916 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" Oct 30 09:55:48.013282 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:48.013245 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d409189c-887a-4f67-a7d4-acf3ef0273c8-kube-api-access-b876x\" (UniqueName: \"kubernetes.io/projected/d409189c-887a-4f67-a7d4-acf3ef0273c8-kube-api-access-b876x\") pod \"d409189c-887a-4f67-a7d4-acf3ef0273c8\" (UID: \"d409189c-887a-4f67-a7d4-acf3ef0273c8\") " Oct 30 09:55:48.013481 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:48.013331 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d409189c-887a-4f67-a7d4-acf3ef0273c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d409189c-887a-4f67-a7d4-acf3ef0273c8-maas-api-openshift-ca\") pod \"d409189c-887a-4f67-a7d4-acf3ef0273c8\" (UID: \"d409189c-887a-4f67-a7d4-acf3ef0273c8\") " Oct 30 09:55:48.013555 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:48.013501 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d409189c-887a-4f67-a7d4-acf3ef0273c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d409189c-887a-4f67-a7d4-acf3ef0273c8-maas-api-xks-ca\") pod \"d409189c-887a-4f67-a7d4-acf3ef0273c8\" (UID: \"d409189c-887a-4f67-a7d4-acf3ef0273c8\") " Oct 30 09:55:48.014204 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:48.014176 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d409189c-887a-4f67-a7d4-acf3ef0273c8-maas-api-openshift-ca" pod "d409189c-887a-4f67-a7d4-acf3ef0273c8" (UID: "d409189c-887a-4f67-a7d4-acf3ef0273c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:48.015803 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:48.015775 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d409189c-887a-4f67-a7d4-acf3ef0273c8-kube-api-access-b876x" pod "d409189c-887a-4f67-a7d4-acf3ef0273c8" (UID: "d409189c-887a-4f67-a7d4-acf3ef0273c8"). InnerVolumeSpecName "kube-api-access-b876x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:48.016638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:48.016569 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d409189c-887a-4f67-a7d4-acf3ef0273c8-maas-api-xks-ca" pod "d409189c-887a-4f67-a7d4-acf3ef0273c8" (UID: "d409189c-887a-4f67-a7d4-acf3ef0273c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:55:48.115375 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:48.115354 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d409189c-887a-4f67-a7d4-acf3ef0273c8-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:48.115540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:48.115378 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b876x\" (UniqueName: \"kubernetes.io/projected/d409189c-887a-4f67-a7d4-acf3ef0273c8-kube-api-access-b876x\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:48.115540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:48.115388 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d409189c-887a-4f67-a7d4-acf3ef0273c8-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:48.804667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:48.804638 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" Oct 30 09:55:48.804847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:48.804631 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5" event={"ID":"d409189c-887a-4f67-a7d4-acf3ef0273c8","Type":"ContainerDied","Data":"9581d05937a6013fcb3893b4070a820164bcd5831e3a7448ec0df9d2c3bcd3b2"} Oct 30 09:55:48.804847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:48.804789 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9581d05937a6013fcb3893b4070a820164bcd5831e3a7448ec0df9d2c3bcd3b2" Oct 30 09:55:49.255791 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.255761 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:55:49.336499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.336403 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-workload-certs\") pod \"df118d0b-7613-4edf-aaa4-af1944531fa6\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " Oct 30 09:55:49.336499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.336448 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-token\" (UniqueName: \"kubernetes.io/projected/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-token\") pod \"df118d0b-7613-4edf-aaa4-af1944531fa6\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " Oct 30 09:55:49.336499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.336466 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-workload-socket\") pod \"df118d0b-7613-4edf-aaa4-af1944531fa6\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " Oct 30 09:55:49.336869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.336535 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-envoy\") pod \"df118d0b-7613-4edf-aaa4-af1944531fa6\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " Oct 30 09:55:49.336869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.336690 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-credential-socket\") pod \"df118d0b-7613-4edf-aaa4-af1944531fa6\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " Oct 30 09:55:49.336869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.336739 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-podinfo\") pod \"df118d0b-7613-4edf-aaa4-af1944531fa6\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " Oct 30 09:55:49.336869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.336833 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-data\") pod \"df118d0b-7613-4edf-aaa4-af1944531fa6\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " Oct 30 09:55:49.337101 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.336878 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df118d0b-7613-4edf-aaa4-af1944531fa6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df118d0b-7613-4edf-aaa4-af1944531fa6-istiod-ca-cert\") pod \"df118d0b-7613-4edf-aaa4-af1944531fa6\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " Oct 30 09:55:49.337101 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.336917 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df118d0b-7613-4edf-aaa4-af1944531fa6-kube-api-access-hxdn4\" (UniqueName: \"kubernetes.io/projected/df118d0b-7613-4edf-aaa4-af1944531fa6-kube-api-access-hxdn4\") pod \"df118d0b-7613-4edf-aaa4-af1944531fa6\" (UID: \"df118d0b-7613-4edf-aaa4-af1944531fa6\") " Oct 30 09:55:49.337208 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.337149 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-data" pod "df118d0b-7613-4edf-aaa4-af1944531fa6" (UID: "df118d0b-7613-4edf-aaa4-af1944531fa6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:49.337269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.337207 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-workload-socket" pod "df118d0b-7613-4edf-aaa4-af1944531fa6" (UID: "df118d0b-7613-4edf-aaa4-af1944531fa6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:49.337502 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.337354 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df118d0b-7613-4edf-aaa4-af1944531fa6-istiod-ca-cert" pod "df118d0b-7613-4edf-aaa4-af1944531fa6" (UID: "df118d0b-7613-4edf-aaa4-af1944531fa6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:49.337502 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.337460 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-workload-certs" pod "df118d0b-7613-4edf-aaa4-af1944531fa6" (UID: "df118d0b-7613-4edf-aaa4-af1944531fa6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:49.337745 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.337538 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-credential-socket" pod "df118d0b-7613-4edf-aaa4-af1944531fa6" (UID: "df118d0b-7613-4edf-aaa4-af1944531fa6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:49.338100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.337840 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.338100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.337869 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df118d0b-7613-4edf-aaa4-af1944531fa6-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.338100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.337883 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.338100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.337897 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.338100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.337913 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.339977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.339955 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-podinfo" pod "df118d0b-7613-4edf-aaa4-af1944531fa6" (UID: "df118d0b-7613-4edf-aaa4-af1944531fa6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:55:49.340182 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.340160 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df118d0b-7613-4edf-aaa4-af1944531fa6-kube-api-access-hxdn4" pod "df118d0b-7613-4edf-aaa4-af1944531fa6" (UID: "df118d0b-7613-4edf-aaa4-af1944531fa6"). InnerVolumeSpecName "kube-api-access-hxdn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:49.340366 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.340341 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-envoy" pod "df118d0b-7613-4edf-aaa4-af1944531fa6" (UID: "df118d0b-7613-4edf-aaa4-af1944531fa6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:49.340684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.340664 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-token" pod "df118d0b-7613-4edf-aaa4-af1944531fa6" (UID: "df118d0b-7613-4edf-aaa4-af1944531fa6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:49.439442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.439411 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.439442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.439437 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.439442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.439447 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df118d0b-7613-4edf-aaa4-af1944531fa6-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.439717 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.439456 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxdn4\" (UniqueName: \"kubernetes.io/projected/df118d0b-7613-4edf-aaa4-af1944531fa6-kube-api-access-hxdn4\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.813531 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.813498 5938 generic.go:356] "Generic (PLEG): container finished" podID="df118d0b-7613-4edf-aaa4-af1944531fa6" containerID="e6304487555bea67797bf74ea39ed434367bb1dfb0daa7232cb858002603102e" exitCode=0 Oct 30 09:55:49.813738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.813544 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" Oct 30 09:55:49.813738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.813617 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" event={"ID":"df118d0b-7613-4edf-aaa4-af1944531fa6","Type":"ContainerDied","Data":"e6304487555bea67797bf74ea39ed434367bb1dfb0daa7232cb858002603102e"} Oct 30 09:55:49.813738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.813675 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw" event={"ID":"df118d0b-7613-4edf-aaa4-af1944531fa6","Type":"ContainerDied","Data":"5c66bb4b56158cd292228b1d3c96bbf1c2039b1c823e68fdf7fe33cb7196b780"} Oct 30 09:55:49.813738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.813705 5938 scope.go:122] "RemoveContainer" containerID="e6304487555bea67797bf74ea39ed434367bb1dfb0daa7232cb858002603102e" Oct 30 09:55:49.830682 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.830664 5938 scope.go:122] "RemoveContainer" containerID="e6304487555bea67797bf74ea39ed434367bb1dfb0daa7232cb858002603102e" Oct 30 09:55:49.830963 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:55:49.830943 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6304487555bea67797bf74ea39ed434367bb1dfb0daa7232cb858002603102e\": container with ID starting with e6304487555bea67797bf74ea39ed434367bb1dfb0daa7232cb858002603102e not found: ID does not exist" containerID="e6304487555bea67797bf74ea39ed434367bb1dfb0daa7232cb858002603102e" Oct 30 09:55:49.831023 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.830972 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6304487555bea67797bf74ea39ed434367bb1dfb0daa7232cb858002603102e"} err="failed to get container status \"e6304487555bea67797bf74ea39ed434367bb1dfb0daa7232cb858002603102e\": rpc error: code = NotFound desc = could not find container \"e6304487555bea67797bf74ea39ed434367bb1dfb0daa7232cb858002603102e\": container with ID starting with e6304487555bea67797bf74ea39ed434367bb1dfb0daa7232cb858002603102e not found: ID does not exist" Oct 30 09:55:49.869733 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.869710 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw"] Oct 30 09:55:49.883438 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:49.883417 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-kxntw"] Oct 30 09:55:50.257693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:50.257662 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df118d0b-7613-4edf-aaa4-af1944531fa6" path="/var/lib/kubelet/pods/df118d0b-7613-4edf-aaa4-af1944531fa6/volumes" Oct 30 09:55:51.509217 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:51.509186 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh"] Oct 30 09:55:51.543678 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:51.543654 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s"] Oct 30 09:55:53.030392 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:53.030359 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz"] Oct 30 09:55:53.043486 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:53.043461 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a88d7ff0-z6wtz"] Oct 30 09:55:54.258383 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:54.258345 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96f40709-18d8-43c6-a11b-76d5098df362" path="/var/lib/kubelet/pods/96f40709-18d8-43c6-a11b-76d5098df362/volumes" Oct 30 09:55:55.020352 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:55.020317 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww"] Oct 30 09:55:56.510329 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:56.510291 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" podUID="9e09ce8d-bfb1-4729-a557-d1d9685547c5" containerName="payload-processing" containerID="cri-o://93bac2fd84e1433b25e64b5fbf98b618cb52c3439a85fa1641a641ac5ba8541f" gracePeriod=25 Oct 30 09:55:56.544430 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:56.544385 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" podUID="1404655b-b5d3-4fb9-b426-131de90077fc" containerName="payload-pre-processing" containerID="cri-o://8e5e23047d5313c85fab6a6a96afe8b5737f1c3702977dc69b80b41a7055f30a" gracePeriod=25 Oct 30 09:55:57.087051 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:57.087019 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" podUID="1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:57.087225 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:57.087059 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" podUID="0bd2cc71-7527-48bf-a58a-db25175e2621" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:57.397836 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:57.397799 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" podUID="1404655b-b5d3-4fb9-b426-131de90077fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:57.398015 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:55:57.397799 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" podUID="9e09ce8d-bfb1-4729-a557-d1d9685547c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:56:00.021336 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.021265 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" podUID="f8f88266-2ef9-4a30-bfee-89816bc75032" containerName="payload-processing" containerID="cri-o://a864682232606ce6f038dcce87a158ac0681bdaaf3e5284b9fee168064c80438" gracePeriod=25 Oct 30 09:56:00.307911 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.307889 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" Oct 30 09:56:00.491100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.491064 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f88266-2ef9-4a30-bfee-89816bc75032-kube-api-access-nqppz\" (UniqueName: \"kubernetes.io/projected/f8f88266-2ef9-4a30-bfee-89816bc75032-kube-api-access-nqppz\") pod \"f8f88266-2ef9-4a30-bfee-89816bc75032\" (UID: \"f8f88266-2ef9-4a30-bfee-89816bc75032\") " Oct 30 09:56:00.491293 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.491114 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f88266-2ef9-4a30-bfee-89816bc75032-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f88266-2ef9-4a30-bfee-89816bc75032-plugins-config-volume\") pod \"f8f88266-2ef9-4a30-bfee-89816bc75032\" (UID: \"f8f88266-2ef9-4a30-bfee-89816bc75032\") " Oct 30 09:56:00.491508 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.491484 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f88266-2ef9-4a30-bfee-89816bc75032-plugins-config-volume" pod "f8f88266-2ef9-4a30-bfee-89816bc75032" (UID: "f8f88266-2ef9-4a30-bfee-89816bc75032"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:00.491870 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.491851 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f88266-2ef9-4a30-bfee-89816bc75032-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:00.493372 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.493346 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f88266-2ef9-4a30-bfee-89816bc75032-kube-api-access-nqppz" pod "f8f88266-2ef9-4a30-bfee-89816bc75032" (UID: "f8f88266-2ef9-4a30-bfee-89816bc75032"). InnerVolumeSpecName "kube-api-access-nqppz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:00.593628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.593562 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqppz\" (UniqueName: \"kubernetes.io/projected/f8f88266-2ef9-4a30-bfee-89816bc75032-kube-api-access-nqppz\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:00.904679 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.904645 5938 generic.go:356] "Generic (PLEG): container finished" podID="f8f88266-2ef9-4a30-bfee-89816bc75032" containerID="a864682232606ce6f038dcce87a158ac0681bdaaf3e5284b9fee168064c80438" exitCode=0 Oct 30 09:56:00.904888 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.904684 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" Oct 30 09:56:00.904888 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.904720 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" event={"ID":"f8f88266-2ef9-4a30-bfee-89816bc75032","Type":"ContainerDied","Data":"a864682232606ce6f038dcce87a158ac0681bdaaf3e5284b9fee168064c80438"} Oct 30 09:56:00.904888 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.904780 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww" event={"ID":"f8f88266-2ef9-4a30-bfee-89816bc75032","Type":"ContainerDied","Data":"0c8ef30f6a8092a8eff2fecebae3dcc9d8e7bf127e106057d56b21f31c6cdabb"} Oct 30 09:56:00.904888 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.904810 5938 scope.go:122] "RemoveContainer" containerID="a864682232606ce6f038dcce87a158ac0681bdaaf3e5284b9fee168064c80438" Oct 30 09:56:00.916037 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.916021 5938 scope.go:122] "RemoveContainer" containerID="a864682232606ce6f038dcce87a158ac0681bdaaf3e5284b9fee168064c80438" Oct 30 09:56:00.916318 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:56:00.916297 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a864682232606ce6f038dcce87a158ac0681bdaaf3e5284b9fee168064c80438\": container with ID starting with a864682232606ce6f038dcce87a158ac0681bdaaf3e5284b9fee168064c80438 not found: ID does not exist" containerID="a864682232606ce6f038dcce87a158ac0681bdaaf3e5284b9fee168064c80438" Oct 30 09:56:00.916376 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.916326 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a864682232606ce6f038dcce87a158ac0681bdaaf3e5284b9fee168064c80438"} err="failed to get container status \"a864682232606ce6f038dcce87a158ac0681bdaaf3e5284b9fee168064c80438\": rpc error: code = NotFound desc = could not find container \"a864682232606ce6f038dcce87a158ac0681bdaaf3e5284b9fee168064c80438\": container with ID starting with a864682232606ce6f038dcce87a158ac0681bdaaf3e5284b9fee168064c80438 not found: ID does not exist" Oct 30 09:56:00.961437 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.961416 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww"] Oct 30 09:56:00.974685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:00.974661 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-304459-d7b764864-zwtww"] Oct 30 09:56:02.257829 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:02.257797 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f88266-2ef9-4a30-bfee-89816bc75032" path="/var/lib/kubelet/pods/f8f88266-2ef9-4a30-bfee-89816bc75032/volumes" Oct 30 09:56:03.520006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.519973 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" Oct 30 09:56:03.531142 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.531121 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd2cc71-7527-48bf-a58a-db25175e2621-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd2cc71-7527-48bf-a58a-db25175e2621-plugins-config-volume\") pod \"0bd2cc71-7527-48bf-a58a-db25175e2621\" (UID: \"0bd2cc71-7527-48bf-a58a-db25175e2621\") " Oct 30 09:56:03.531285 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.531171 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd2cc71-7527-48bf-a58a-db25175e2621-kube-api-access-7l9dx\" (UniqueName: \"kubernetes.io/projected/0bd2cc71-7527-48bf-a58a-db25175e2621-kube-api-access-7l9dx\") pod \"0bd2cc71-7527-48bf-a58a-db25175e2621\" (UID: \"0bd2cc71-7527-48bf-a58a-db25175e2621\") " Oct 30 09:56:03.531726 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.531702 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd2cc71-7527-48bf-a58a-db25175e2621-plugins-config-volume" pod "0bd2cc71-7527-48bf-a58a-db25175e2621" (UID: "0bd2cc71-7527-48bf-a58a-db25175e2621"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:03.532025 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.532004 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd2cc71-7527-48bf-a58a-db25175e2621-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:03.533965 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.533937 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd2cc71-7527-48bf-a58a-db25175e2621-kube-api-access-7l9dx" pod "0bd2cc71-7527-48bf-a58a-db25175e2621" (UID: "0bd2cc71-7527-48bf-a58a-db25175e2621"). InnerVolumeSpecName "kube-api-access-7l9dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:03.571021 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.570997 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" Oct 30 09:56:03.633490 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.633462 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24-plugins-config-volume\") pod \"1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24\" (UID: \"1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24\") " Oct 30 09:56:03.633672 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.633515 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24-kube-api-access-7qljf\" (UniqueName: \"kubernetes.io/projected/1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24-kube-api-access-7qljf\") pod \"1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24\" (UID: \"1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24\") " Oct 30 09:56:03.633838 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.633816 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24-plugins-config-volume" pod "1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24" (UID: "1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:03.634071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.634043 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l9dx\" (UniqueName: \"kubernetes.io/projected/0bd2cc71-7527-48bf-a58a-db25175e2621-kube-api-access-7l9dx\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:03.634071 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.634070 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:03.635859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.635832 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24-kube-api-access-7qljf" pod "1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24" (UID: "1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24"). InnerVolumeSpecName "kube-api-access-7qljf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:03.735025 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.734954 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qljf\" (UniqueName: \"kubernetes.io/projected/1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24-kube-api-access-7qljf\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:03.930435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.930404 5938 generic.go:356] "Generic (PLEG): container finished" podID="0bd2cc71-7527-48bf-a58a-db25175e2621" containerID="e752ab005a48d54d767a016ecaf3713df0512be0508bacef957b0d1578b78756" exitCode=0 Oct 30 09:56:03.930642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.930461 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" Oct 30 09:56:03.930642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.930490 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" event={"ID":"0bd2cc71-7527-48bf-a58a-db25175e2621","Type":"ContainerDied","Data":"e752ab005a48d54d767a016ecaf3713df0512be0508bacef957b0d1578b78756"} Oct 30 09:56:03.930642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.930598 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87" event={"ID":"0bd2cc71-7527-48bf-a58a-db25175e2621","Type":"ContainerDied","Data":"e7ff73af9807164d99f14889af92d4845acf5d8c55508307739c5eff5dd123ff"} Oct 30 09:56:03.930642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.930635 5938 scope.go:122] "RemoveContainer" containerID="e752ab005a48d54d767a016ecaf3713df0512be0508bacef957b0d1578b78756" Oct 30 09:56:03.932049 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.932024 5938 generic.go:356] "Generic (PLEG): container finished" podID="1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24" containerID="ba9e7c5ed8cc7393bfdcc1ceb208605fe092c6e70553aa0b93330638f77427b3" exitCode=0 Oct 30 09:56:03.932178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.932058 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" Oct 30 09:56:03.932178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.932098 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" event={"ID":"1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24","Type":"ContainerDied","Data":"ba9e7c5ed8cc7393bfdcc1ceb208605fe092c6e70553aa0b93330638f77427b3"} Oct 30 09:56:03.932178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.932139 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq" event={"ID":"1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24","Type":"ContainerDied","Data":"7d03422e06d989db1844e9f9bdee57a177eb57213ebf183b166ae319f32c4dde"} Oct 30 09:56:03.941919 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.941891 5938 scope.go:122] "RemoveContainer" containerID="e752ab005a48d54d767a016ecaf3713df0512be0508bacef957b0d1578b78756" Oct 30 09:56:03.942199 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:56:03.942180 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e752ab005a48d54d767a016ecaf3713df0512be0508bacef957b0d1578b78756\": container with ID starting with e752ab005a48d54d767a016ecaf3713df0512be0508bacef957b0d1578b78756 not found: ID does not exist" containerID="e752ab005a48d54d767a016ecaf3713df0512be0508bacef957b0d1578b78756" Oct 30 09:56:03.942275 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.942208 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e752ab005a48d54d767a016ecaf3713df0512be0508bacef957b0d1578b78756"} err="failed to get container status \"e752ab005a48d54d767a016ecaf3713df0512be0508bacef957b0d1578b78756\": rpc error: code = NotFound desc = could not find container \"e752ab005a48d54d767a016ecaf3713df0512be0508bacef957b0d1578b78756\": container with ID starting with e752ab005a48d54d767a016ecaf3713df0512be0508bacef957b0d1578b78756 not found: ID does not exist" Oct 30 09:56:03.942275 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.942223 5938 scope.go:122] "RemoveContainer" containerID="ba9e7c5ed8cc7393bfdcc1ceb208605fe092c6e70553aa0b93330638f77427b3" Oct 30 09:56:03.951563 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.951545 5938 scope.go:122] "RemoveContainer" containerID="ba9e7c5ed8cc7393bfdcc1ceb208605fe092c6e70553aa0b93330638f77427b3" Oct 30 09:56:03.951866 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:56:03.951845 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9e7c5ed8cc7393bfdcc1ceb208605fe092c6e70553aa0b93330638f77427b3\": container with ID starting with ba9e7c5ed8cc7393bfdcc1ceb208605fe092c6e70553aa0b93330638f77427b3 not found: ID does not exist" containerID="ba9e7c5ed8cc7393bfdcc1ceb208605fe092c6e70553aa0b93330638f77427b3" Oct 30 09:56:03.951939 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.951872 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9e7c5ed8cc7393bfdcc1ceb208605fe092c6e70553aa0b93330638f77427b3"} err="failed to get container status \"ba9e7c5ed8cc7393bfdcc1ceb208605fe092c6e70553aa0b93330638f77427b3\": rpc error: code = NotFound desc = could not find container \"ba9e7c5ed8cc7393bfdcc1ceb208605fe092c6e70553aa0b93330638f77427b3\": container with ID starting with ba9e7c5ed8cc7393bfdcc1ceb208605fe092c6e70553aa0b93330638f77427b3 not found: ID does not exist" Oct 30 09:56:03.986256 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.986205 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87"] Oct 30 09:56:03.999568 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:03.999547 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-aa2fdf-66dc96d4c6-8vn87"] Oct 30 09:56:04.035240 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:04.035202 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq"] Oct 30 09:56:04.039200 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:04.039180 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa2fdf-65fc775d87-2n8sq"] Oct 30 09:56:04.259040 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:04.258978 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd2cc71-7527-48bf-a58a-db25175e2621" path="/var/lib/kubelet/pods/0bd2cc71-7527-48bf-a58a-db25175e2621/volumes" Oct 30 09:56:04.259393 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:04.259381 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24" path="/var/lib/kubelet/pods/1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24/volumes" Oct 30 09:56:07.397014 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:07.396978 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" podUID="9e09ce8d-bfb1-4729-a557-d1d9685547c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:56:07.397460 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:07.397027 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" podUID="1404655b-b5d3-4fb9-b426-131de90077fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:56:11.224148 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:11.224113 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c"] Oct 30 09:56:11.237321 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:11.237291 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-aa2fdf-sjp7c"] Oct 30 09:56:11.802899 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:11.802866 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk"] Oct 30 09:56:11.815920 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:11.815891 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb1a2a-n6fpk"] Oct 30 09:56:12.258772 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:12.258740 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66cc0994-09e1-4863-b875-9451b5b7f1b5" path="/var/lib/kubelet/pods/66cc0994-09e1-4863-b875-9451b5b7f1b5/volumes" Oct 30 09:56:12.259174 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:12.259160 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9596f991-d4dc-469a-bf22-b87b5a3f47a2" path="/var/lib/kubelet/pods/9596f991-d4dc-469a-bf22-b87b5a3f47a2/volumes" Oct 30 09:56:15.808975 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:56:15.808944 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:56:16.840443 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:16.840415 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" Oct 30 09:56:16.896941 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:16.896911 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" Oct 30 09:56:16.910529 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:16.910500 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e09ce8d-bfb1-4729-a557-d1d9685547c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e09ce8d-bfb1-4729-a557-d1d9685547c5-plugins-config-volume\") pod \"9e09ce8d-bfb1-4729-a557-d1d9685547c5\" (UID: \"9e09ce8d-bfb1-4729-a557-d1d9685547c5\") " Oct 30 09:56:16.910708 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:16.910534 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e09ce8d-bfb1-4729-a557-d1d9685547c5-kube-api-access-d5zjn\" (UniqueName: \"kubernetes.io/projected/9e09ce8d-bfb1-4729-a557-d1d9685547c5-kube-api-access-d5zjn\") pod \"9e09ce8d-bfb1-4729-a557-d1d9685547c5\" (UID: \"9e09ce8d-bfb1-4729-a557-d1d9685547c5\") " Oct 30 09:56:16.910857 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:16.910836 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e09ce8d-bfb1-4729-a557-d1d9685547c5-plugins-config-volume" pod "9e09ce8d-bfb1-4729-a557-d1d9685547c5" (UID: "9e09ce8d-bfb1-4729-a557-d1d9685547c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:16.911035 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:16.911018 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e09ce8d-bfb1-4729-a557-d1d9685547c5-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:16.912835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:16.912814 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e09ce8d-bfb1-4729-a557-d1d9685547c5-kube-api-access-d5zjn" pod "9e09ce8d-bfb1-4729-a557-d1d9685547c5" (UID: "9e09ce8d-bfb1-4729-a557-d1d9685547c5"). InnerVolumeSpecName "kube-api-access-d5zjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:17.012350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.012322 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1404655b-b5d3-4fb9-b426-131de90077fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1404655b-b5d3-4fb9-b426-131de90077fc-plugins-config-volume\") pod \"1404655b-b5d3-4fb9-b426-131de90077fc\" (UID: \"1404655b-b5d3-4fb9-b426-131de90077fc\") " Oct 30 09:56:17.012506 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.012407 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1404655b-b5d3-4fb9-b426-131de90077fc-kube-api-access-n6kh9\" (UniqueName: \"kubernetes.io/projected/1404655b-b5d3-4fb9-b426-131de90077fc-kube-api-access-n6kh9\") pod \"1404655b-b5d3-4fb9-b426-131de90077fc\" (UID: \"1404655b-b5d3-4fb9-b426-131de90077fc\") " Oct 30 09:56:17.012756 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.012735 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1404655b-b5d3-4fb9-b426-131de90077fc-plugins-config-volume" pod "1404655b-b5d3-4fb9-b426-131de90077fc" (UID: "1404655b-b5d3-4fb9-b426-131de90077fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:17.013044 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.013028 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5zjn\" (UniqueName: \"kubernetes.io/projected/9e09ce8d-bfb1-4729-a557-d1d9685547c5-kube-api-access-d5zjn\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:17.013085 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.013047 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1404655b-b5d3-4fb9-b426-131de90077fc-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:17.014534 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.014512 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1404655b-b5d3-4fb9-b426-131de90077fc-kube-api-access-n6kh9" pod "1404655b-b5d3-4fb9-b426-131de90077fc" (UID: "1404655b-b5d3-4fb9-b426-131de90077fc"). InnerVolumeSpecName "kube-api-access-n6kh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:17.033768 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.033744 5938 generic.go:356] "Generic (PLEG): container finished" podID="9e09ce8d-bfb1-4729-a557-d1d9685547c5" containerID="93bac2fd84e1433b25e64b5fbf98b618cb52c3439a85fa1641a641ac5ba8541f" exitCode=0 Oct 30 09:56:17.033881 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.033795 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" Oct 30 09:56:17.033881 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.033822 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" event={"ID":"9e09ce8d-bfb1-4729-a557-d1d9685547c5","Type":"ContainerDied","Data":"93bac2fd84e1433b25e64b5fbf98b618cb52c3439a85fa1641a641ac5ba8541f"} Oct 30 09:56:17.033988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.033893 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh" event={"ID":"9e09ce8d-bfb1-4729-a557-d1d9685547c5","Type":"ContainerDied","Data":"6a30c42783b27a3ca4ca4a80116978ab9aa717b892605cd2543d852bdeb71c99"} Oct 30 09:56:17.033988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.033905 5938 scope.go:122] "RemoveContainer" containerID="93bac2fd84e1433b25e64b5fbf98b618cb52c3439a85fa1641a641ac5ba8541f" Oct 30 09:56:17.035545 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.035521 5938 generic.go:356] "Generic (PLEG): container finished" podID="1404655b-b5d3-4fb9-b426-131de90077fc" containerID="8e5e23047d5313c85fab6a6a96afe8b5737f1c3702977dc69b80b41a7055f30a" exitCode=0 Oct 30 09:56:17.035697 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.035567 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" Oct 30 09:56:17.035697 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.035650 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" event={"ID":"1404655b-b5d3-4fb9-b426-131de90077fc","Type":"ContainerDied","Data":"8e5e23047d5313c85fab6a6a96afe8b5737f1c3702977dc69b80b41a7055f30a"} Oct 30 09:56:17.035844 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.035701 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s" event={"ID":"1404655b-b5d3-4fb9-b426-131de90077fc","Type":"ContainerDied","Data":"1127c5c85eca497587609044ffb55307570195f26ee580fb7bb0f333ee2cc1a9"} Oct 30 09:56:17.045293 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.045275 5938 scope.go:122] "RemoveContainer" containerID="93bac2fd84e1433b25e64b5fbf98b618cb52c3439a85fa1641a641ac5ba8541f" Oct 30 09:56:17.045552 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:56:17.045533 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93bac2fd84e1433b25e64b5fbf98b618cb52c3439a85fa1641a641ac5ba8541f\": container with ID starting with 93bac2fd84e1433b25e64b5fbf98b618cb52c3439a85fa1641a641ac5ba8541f not found: ID does not exist" containerID="93bac2fd84e1433b25e64b5fbf98b618cb52c3439a85fa1641a641ac5ba8541f" Oct 30 09:56:17.045673 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.045564 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93bac2fd84e1433b25e64b5fbf98b618cb52c3439a85fa1641a641ac5ba8541f"} err="failed to get container status \"93bac2fd84e1433b25e64b5fbf98b618cb52c3439a85fa1641a641ac5ba8541f\": rpc error: code = NotFound desc = could not find container \"93bac2fd84e1433b25e64b5fbf98b618cb52c3439a85fa1641a641ac5ba8541f\": container with ID starting with 93bac2fd84e1433b25e64b5fbf98b618cb52c3439a85fa1641a641ac5ba8541f not found: ID does not exist" Oct 30 09:56:17.045673 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.045612 5938 scope.go:122] "RemoveContainer" containerID="8e5e23047d5313c85fab6a6a96afe8b5737f1c3702977dc69b80b41a7055f30a" Oct 30 09:56:17.055174 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.055152 5938 scope.go:122] "RemoveContainer" containerID="8e5e23047d5313c85fab6a6a96afe8b5737f1c3702977dc69b80b41a7055f30a" Oct 30 09:56:17.055431 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:56:17.055411 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e5e23047d5313c85fab6a6a96afe8b5737f1c3702977dc69b80b41a7055f30a\": container with ID starting with 8e5e23047d5313c85fab6a6a96afe8b5737f1c3702977dc69b80b41a7055f30a not found: ID does not exist" containerID="8e5e23047d5313c85fab6a6a96afe8b5737f1c3702977dc69b80b41a7055f30a" Oct 30 09:56:17.055488 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.055438 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5e23047d5313c85fab6a6a96afe8b5737f1c3702977dc69b80b41a7055f30a"} err="failed to get container status \"8e5e23047d5313c85fab6a6a96afe8b5737f1c3702977dc69b80b41a7055f30a\": rpc error: code = NotFound desc = could not find container \"8e5e23047d5313c85fab6a6a96afe8b5737f1c3702977dc69b80b41a7055f30a\": container with ID starting with 8e5e23047d5313c85fab6a6a96afe8b5737f1c3702977dc69b80b41a7055f30a not found: ID does not exist" Oct 30 09:56:17.095654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.095623 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s"] Oct 30 09:56:17.096805 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.096784 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb1a2a-6857496555-hzg9s"] Oct 30 09:56:17.114144 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.114118 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6kh9\" (UniqueName: \"kubernetes.io/projected/1404655b-b5d3-4fb9-b426-131de90077fc-kube-api-access-n6kh9\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:17.132910 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.132889 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh"] Oct 30 09:56:17.136149 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:17.136128 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb1a2a-766d5fd4f8-tntkh"] Oct 30 09:56:18.257550 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:18.257509 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1404655b-b5d3-4fb9-b426-131de90077fc" path="/var/lib/kubelet/pods/1404655b-b5d3-4fb9-b426-131de90077fc/volumes" Oct 30 09:56:18.258141 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:18.258120 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e09ce8d-bfb1-4729-a557-d1d9685547c5" path="/var/lib/kubelet/pods/9e09ce8d-bfb1-4729-a557-d1d9685547c5/volumes" Oct 30 09:56:18.690703 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:18.686338 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr"] Oct 30 09:56:18.690703 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:18.686668 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" podUID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" containerName="istio-proxy" containerID="cri-o://c36ecb0d6a73b3279f6399132375799c3fbbd2e13178ee037eb117550b310582" gracePeriod=30 Oct 30 09:56:19.201136 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201105 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h"] Oct 30 09:56:19.201800 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201783 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b9985d34-9195-47d7-8e4c-fb5a87ca969a" containerName="istio-proxy" Oct 30 09:56:19.201800 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201801 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9985d34-9195-47d7-8e4c-fb5a87ca969a" containerName="istio-proxy" Oct 30 09:56:19.201934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201814 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b9985d34-9195-47d7-8e4c-fb5a87ca969a" containerName="istio-proxy" Oct 30 09:56:19.201934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201820 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9985d34-9195-47d7-8e4c-fb5a87ca969a" containerName="istio-proxy" Oct 30 09:56:19.201934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201833 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f8f88266-2ef9-4a30-bfee-89816bc75032" containerName="payload-processing" Oct 30 09:56:19.201934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201838 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f88266-2ef9-4a30-bfee-89816bc75032" containerName="payload-processing" Oct 30 09:56:19.201934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201852 5938 cpu_manager.go:414] "containerMap: removing container" podUID="df118d0b-7613-4edf-aaa4-af1944531fa6" containerName="istio-proxy" Oct 30 09:56:19.201934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201856 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df118d0b-7613-4edf-aaa4-af1944531fa6" containerName="istio-proxy" Oct 30 09:56:19.201934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201869 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1404655b-b5d3-4fb9-b426-131de90077fc" containerName="payload-pre-processing" Oct 30 09:56:19.201934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201875 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1404655b-b5d3-4fb9-b426-131de90077fc" containerName="payload-pre-processing" Oct 30 09:56:19.201934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201889 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24" containerName="payload-pre-processing" Oct 30 09:56:19.201934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201893 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24" containerName="payload-pre-processing" Oct 30 09:56:19.201934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201901 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d409189c-887a-4f67-a7d4-acf3ef0273c8" containerName="revoke-keys" Oct 30 09:56:19.201934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201906 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d409189c-887a-4f67-a7d4-acf3ef0273c8" containerName="revoke-keys" Oct 30 09:56:19.201934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201918 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0bd2cc71-7527-48bf-a58a-db25175e2621" containerName="payload-processing" Oct 30 09:56:19.201934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201922 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd2cc71-7527-48bf-a58a-db25175e2621" containerName="payload-processing" Oct 30 09:56:19.201934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201931 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9e09ce8d-bfb1-4729-a557-d1d9685547c5" containerName="payload-processing" Oct 30 09:56:19.201934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.201935 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e09ce8d-bfb1-4729-a557-d1d9685547c5" containerName="payload-processing" Oct 30 09:56:19.202492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.202029 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9985d34-9195-47d7-8e4c-fb5a87ca969a" containerName="istio-proxy" Oct 30 09:56:19.202492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.202038 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9985d34-9195-47d7-8e4c-fb5a87ca969a" containerName="istio-proxy" Oct 30 09:56:19.202492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.202044 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f88266-2ef9-4a30-bfee-89816bc75032" containerName="payload-processing" Oct 30 09:56:19.202492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.202052 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="df118d0b-7613-4edf-aaa4-af1944531fa6" containerName="istio-proxy" Oct 30 09:56:19.202492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.202060 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1404655b-b5d3-4fb9-b426-131de90077fc" containerName="payload-pre-processing" Oct 30 09:56:19.202492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.202068 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2e5d9f-2aeb-4e7f-b0f4-72e4f4afab24" containerName="payload-pre-processing" Oct 30 09:56:19.202492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.202072 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d409189c-887a-4f67-a7d4-acf3ef0273c8" containerName="revoke-keys" Oct 30 09:56:19.202492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.202079 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd2cc71-7527-48bf-a58a-db25175e2621" containerName="payload-processing" Oct 30 09:56:19.202492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.202085 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e09ce8d-bfb1-4729-a557-d1d9685547c5" containerName="payload-processing" Oct 30 09:56:19.206884 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.206867 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h" Oct 30 09:56:19.225806 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.225783 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h"] Oct 30 09:56:19.236062 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.236018 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:56:19.236374 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.236318 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:56:19.351061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.351031 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw9ww\" (UniqueName: \"kubernetes.io/projected/2f722682-e624-46fe-81f0-c630c2718a54-kube-api-access-rw9ww\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h\" (UID: \"2f722682-e624-46fe-81f0-c630c2718a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h" Oct 30 09:56:19.351419 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.351161 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f722682-e624-46fe-81f0-c630c2718a54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h\" (UID: \"2f722682-e624-46fe-81f0-c630c2718a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h" Oct 30 09:56:19.351419 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.351232 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f722682-e624-46fe-81f0-c630c2718a54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h\" (UID: \"2f722682-e624-46fe-81f0-c630c2718a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h" Oct 30 09:56:19.394740 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.394705 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd"] Oct 30 09:56:19.399080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.399060 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd" Oct 30 09:56:19.419825 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.419806 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd"] Oct 30 09:56:19.452684 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.452627 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw9ww\" (UniqueName: \"kubernetes.io/projected/2f722682-e624-46fe-81f0-c630c2718a54-kube-api-access-rw9ww\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h\" (UID: \"2f722682-e624-46fe-81f0-c630c2718a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h" Oct 30 09:56:19.452833 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.452815 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f722682-e624-46fe-81f0-c630c2718a54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h\" (UID: \"2f722682-e624-46fe-81f0-c630c2718a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h" Oct 30 09:56:19.452932 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.452915 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f722682-e624-46fe-81f0-c630c2718a54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h\" (UID: \"2f722682-e624-46fe-81f0-c630c2718a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h" Oct 30 09:56:19.453483 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.453460 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f722682-e624-46fe-81f0-c630c2718a54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h\" (UID: \"2f722682-e624-46fe-81f0-c630c2718a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h" Oct 30 09:56:19.455435 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.455412 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f722682-e624-46fe-81f0-c630c2718a54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h\" (UID: \"2f722682-e624-46fe-81f0-c630c2718a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h" Oct 30 09:56:19.473174 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.473156 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw9ww\" (UniqueName: \"kubernetes.io/projected/2f722682-e624-46fe-81f0-c630c2718a54-kube-api-access-rw9ww\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h\" (UID: \"2f722682-e624-46fe-81f0-c630c2718a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h" Oct 30 09:56:19.520780 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.520760 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h" Oct 30 09:56:19.554206 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.554183 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c62c859-8cd1-478e-8f0c-d67506b620cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd\" (UID: \"9c62c859-8cd1-478e-8f0c-d67506b620cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd" Oct 30 09:56:19.554329 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.554224 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c62c859-8cd1-478e-8f0c-d67506b620cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd\" (UID: \"9c62c859-8cd1-478e-8f0c-d67506b620cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd" Oct 30 09:56:19.554388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.554350 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcvkl\" (UniqueName: \"kubernetes.io/projected/9c62c859-8cd1-478e-8f0c-d67506b620cd-kube-api-access-mcvkl\") pod \"maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd\" (UID: \"9c62c859-8cd1-478e-8f0c-d67506b620cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd" Oct 30 09:56:19.655408 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.655373 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c62c859-8cd1-478e-8f0c-d67506b620cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd\" (UID: \"9c62c859-8cd1-478e-8f0c-d67506b620cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd" Oct 30 09:56:19.655408 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.655415 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c62c859-8cd1-478e-8f0c-d67506b620cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd\" (UID: \"9c62c859-8cd1-478e-8f0c-d67506b620cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd" Oct 30 09:56:19.655674 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.655493 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcvkl\" (UniqueName: \"kubernetes.io/projected/9c62c859-8cd1-478e-8f0c-d67506b620cd-kube-api-access-mcvkl\") pod \"maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd\" (UID: \"9c62c859-8cd1-478e-8f0c-d67506b620cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd" Oct 30 09:56:19.656171 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.656145 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c62c859-8cd1-478e-8f0c-d67506b620cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd\" (UID: \"9c62c859-8cd1-478e-8f0c-d67506b620cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd" Oct 30 09:56:19.657937 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.657914 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c62c859-8cd1-478e-8f0c-d67506b620cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd\" (UID: \"9c62c859-8cd1-478e-8f0c-d67506b620cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd" Oct 30 09:56:19.676517 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.676496 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcvkl\" (UniqueName: \"kubernetes.io/projected/9c62c859-8cd1-478e-8f0c-d67506b620cd-kube-api-access-mcvkl\") pod \"maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd\" (UID: \"9c62c859-8cd1-478e-8f0c-d67506b620cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd" Oct 30 09:56:19.713759 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.713710 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd" Oct 30 09:56:19.824455 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:19.824425 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h"] Oct 30 09:56:19.825988 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:56:19.825954 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f722682_e624_46fe_81f0_c630c2718a54.slice/crio-56e09ab2db861171c267498a90d0b1d8a593464776bf7135e691000c1f92c685 WatchSource:0}: Error finding container 56e09ab2db861171c267498a90d0b1d8a593464776bf7135e691000c1f92c685: Status 404 returned error can't find the container with id 56e09ab2db861171c267498a90d0b1d8a593464776bf7135e691000c1f92c685 Oct 30 09:56:20.024991 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:20.024957 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd"] Oct 30 09:56:20.059654 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:20.059622 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd" event={"ID":"9c62c859-8cd1-478e-8f0c-d67506b620cd","Type":"ContainerStarted","Data":"c898fcc76376f915ee1bad6551bd5ba50b40888fd5e64ff70e10b4ee1c6bcf79"} Oct 30 09:56:20.061097 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:20.061064 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h" event={"ID":"2f722682-e624-46fe-81f0-c630c2718a54","Type":"ContainerStarted","Data":"56e09ab2db861171c267498a90d0b1d8a593464776bf7135e691000c1f92c685"} Oct 30 09:56:21.069617 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:21.069536 5938 generic.go:356] "Generic (PLEG): container finished" podID="9c62c859-8cd1-478e-8f0c-d67506b620cd" containerID="0a4a35d4824db862ecdea0cd5f7d3014ff78994fd3666794b160f01a52e7bcc8" exitCode=0 Oct 30 09:56:21.069984 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:21.069629 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd" event={"ID":"9c62c859-8cd1-478e-8f0c-d67506b620cd","Type":"ContainerDied","Data":"0a4a35d4824db862ecdea0cd5f7d3014ff78994fd3666794b160f01a52e7bcc8"} Oct 30 09:56:21.071546 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:21.071521 5938 generic.go:356] "Generic (PLEG): container finished" podID="2f722682-e624-46fe-81f0-c630c2718a54" containerID="b77ee8109f829e99882656d427bf63a8e1956461074907c6ac61c7b556c46f13" exitCode=0 Oct 30 09:56:21.071679 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:21.071559 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h" event={"ID":"2f722682-e624-46fe-81f0-c630c2718a54","Type":"ContainerDied","Data":"b77ee8109f829e99882656d427bf63a8e1956461074907c6ac61c7b556c46f13"} Oct 30 09:56:22.240004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.239981 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd" Oct 30 09:56:22.288052 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.288026 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h" Oct 30 09:56:22.391804 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.391772 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f722682-e624-46fe-81f0-c630c2718a54-kube-api-access-rw9ww\" (UniqueName: \"kubernetes.io/projected/2f722682-e624-46fe-81f0-c630c2718a54-kube-api-access-rw9ww\") pod \"2f722682-e624-46fe-81f0-c630c2718a54\" (UID: \"2f722682-e624-46fe-81f0-c630c2718a54\") " Oct 30 09:56:22.391995 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.391826 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c62c859-8cd1-478e-8f0c-d67506b620cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c62c859-8cd1-478e-8f0c-d67506b620cd-maas-api-openshift-ca\") pod \"9c62c859-8cd1-478e-8f0c-d67506b620cd\" (UID: \"9c62c859-8cd1-478e-8f0c-d67506b620cd\") " Oct 30 09:56:22.391995 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.391905 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c62c859-8cd1-478e-8f0c-d67506b620cd-kube-api-access-mcvkl\" (UniqueName: \"kubernetes.io/projected/9c62c859-8cd1-478e-8f0c-d67506b620cd-kube-api-access-mcvkl\") pod \"9c62c859-8cd1-478e-8f0c-d67506b620cd\" (UID: \"9c62c859-8cd1-478e-8f0c-d67506b620cd\") " Oct 30 09:56:22.391995 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.391989 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f722682-e624-46fe-81f0-c630c2718a54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f722682-e624-46fe-81f0-c630c2718a54-maas-api-xks-ca\") pod \"2f722682-e624-46fe-81f0-c630c2718a54\" (UID: \"2f722682-e624-46fe-81f0-c630c2718a54\") " Oct 30 09:56:22.392195 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.392077 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f722682-e624-46fe-81f0-c630c2718a54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f722682-e624-46fe-81f0-c630c2718a54-maas-api-openshift-ca\") pod \"2f722682-e624-46fe-81f0-c630c2718a54\" (UID: \"2f722682-e624-46fe-81f0-c630c2718a54\") " Oct 30 09:56:22.392195 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.392143 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c62c859-8cd1-478e-8f0c-d67506b620cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c62c859-8cd1-478e-8f0c-d67506b620cd-maas-api-xks-ca\") pod \"9c62c859-8cd1-478e-8f0c-d67506b620cd\" (UID: \"9c62c859-8cd1-478e-8f0c-d67506b620cd\") " Oct 30 09:56:22.392272 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.392249 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c62c859-8cd1-478e-8f0c-d67506b620cd-maas-api-openshift-ca" pod "9c62c859-8cd1-478e-8f0c-d67506b620cd" (UID: "9c62c859-8cd1-478e-8f0c-d67506b620cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:22.393153 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.392804 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c62c859-8cd1-478e-8f0c-d67506b620cd-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:22.393153 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.392822 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f722682-e624-46fe-81f0-c630c2718a54-maas-api-openshift-ca" pod "2f722682-e624-46fe-81f0-c630c2718a54" (UID: "2f722682-e624-46fe-81f0-c630c2718a54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:22.394517 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.394478 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f722682-e624-46fe-81f0-c630c2718a54-kube-api-access-rw9ww" pod "2f722682-e624-46fe-81f0-c630c2718a54" (UID: "2f722682-e624-46fe-81f0-c630c2718a54"). InnerVolumeSpecName "kube-api-access-rw9ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:22.394695 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.394517 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f722682-e624-46fe-81f0-c630c2718a54-maas-api-xks-ca" pod "2f722682-e624-46fe-81f0-c630c2718a54" (UID: "2f722682-e624-46fe-81f0-c630c2718a54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:22.394832 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.394814 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c62c859-8cd1-478e-8f0c-d67506b620cd-kube-api-access-mcvkl" pod "9c62c859-8cd1-478e-8f0c-d67506b620cd" (UID: "9c62c859-8cd1-478e-8f0c-d67506b620cd"). InnerVolumeSpecName "kube-api-access-mcvkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:22.395396 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.395352 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c62c859-8cd1-478e-8f0c-d67506b620cd-maas-api-xks-ca" pod "9c62c859-8cd1-478e-8f0c-d67506b620cd" (UID: "9c62c859-8cd1-478e-8f0c-d67506b620cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:22.493929 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.493906 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcvkl\" (UniqueName: \"kubernetes.io/projected/9c62c859-8cd1-478e-8f0c-d67506b620cd-kube-api-access-mcvkl\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:22.493929 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.493928 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f722682-e624-46fe-81f0-c630c2718a54-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:22.494061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.493938 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f722682-e624-46fe-81f0-c630c2718a54-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:22.494061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.493947 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c62c859-8cd1-478e-8f0c-d67506b620cd-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:22.494061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:22.493955 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw9ww\" (UniqueName: \"kubernetes.io/projected/2f722682-e624-46fe-81f0-c630c2718a54-kube-api-access-rw9ww\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:23.088630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:23.088576 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd" Oct 30 09:56:23.088630 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:23.088617 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd" event={"ID":"9c62c859-8cd1-478e-8f0c-d67506b620cd","Type":"ContainerDied","Data":"c898fcc76376f915ee1bad6551bd5ba50b40888fd5e64ff70e10b4ee1c6bcf79"} Oct 30 09:56:23.088886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:23.088674 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c898fcc76376f915ee1bad6551bd5ba50b40888fd5e64ff70e10b4ee1c6bcf79" Oct 30 09:56:23.090488 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:23.090469 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h" Oct 30 09:56:23.090632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:23.090524 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h" event={"ID":"2f722682-e624-46fe-81f0-c630c2718a54","Type":"ContainerDied","Data":"56e09ab2db861171c267498a90d0b1d8a593464776bf7135e691000c1f92c685"} Oct 30 09:56:23.090632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:23.090601 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56e09ab2db861171c267498a90d0b1d8a593464776bf7135e691000c1f92c685" Oct 30 09:56:23.960620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:23.960548 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:56:24.104345 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.104309 5938 generic.go:356] "Generic (PLEG): container finished" podID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" containerID="c36ecb0d6a73b3279f6399132375799c3fbbd2e13178ee037eb117550b310582" exitCode=0 Oct 30 09:56:24.105003 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.104366 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" Oct 30 09:56:24.105003 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.104358 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" event={"ID":"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe","Type":"ContainerDied","Data":"c36ecb0d6a73b3279f6399132375799c3fbbd2e13178ee037eb117550b310582"} Oct 30 09:56:24.105003 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.104807 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr" event={"ID":"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe","Type":"ContainerDied","Data":"a49207c071e67adc34a19433ec5d1a9fc5fa183ecccbcc1cebd2333d73fdcc1c"} Oct 30 09:56:24.105003 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.104848 5938 scope.go:122] "RemoveContainer" containerID="c36ecb0d6a73b3279f6399132375799c3fbbd2e13178ee037eb117550b310582" Oct 30 09:56:24.116689 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.116655 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-envoy\") pod \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " Oct 30 09:56:24.116839 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.116723 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-credential-socket\") pod \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " Oct 30 09:56:24.116839 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.116769 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-workload-certs\") pod \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " Oct 30 09:56:24.116979 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.116921 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-podinfo\") pod \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " Oct 30 09:56:24.116979 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.116959 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-workload-socket\") pod \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " Oct 30 09:56:24.117100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.116993 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-data\") pod \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " Oct 30 09:56:24.117100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.117067 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-token\" (UniqueName: \"kubernetes.io/projected/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-token\") pod \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " Oct 30 09:56:24.117212 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.117126 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-kube-api-access-h6zrv\" (UniqueName: \"kubernetes.io/projected/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-kube-api-access-h6zrv\") pod \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " Oct 30 09:56:24.117278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.117225 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-workload-certs" pod "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" (UID: "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.117626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.117406 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istiod-ca-cert\") pod \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\" (UID: \"53e745f6-3ef0-4391-b1ec-2e0feb40ddfe\") " Oct 30 09:56:24.117626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.117539 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-credential-socket" pod "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" (UID: "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.118090 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.117978 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-data" pod "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" (UID: "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.118090 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.118041 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istiod-ca-cert" pod "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" (UID: "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:24.118309 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.118250 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-workload-socket" pod "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" (UID: "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.118426 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.118404 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.118528 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.118434 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.118528 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.118451 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.118528 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.118465 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.118528 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.118479 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.118916 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.118801 5938 scope.go:122] "RemoveContainer" containerID="ec34ecee1d481d600b1cd060d276523bc3f8cf8e1526f700e85fa150bf40619d" Oct 30 09:56:24.120313 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.120278 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-envoy" pod "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" (UID: "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.121117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.121091 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-token" pod "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" (UID: "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:24.121386 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.121359 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-kube-api-access-h6zrv" pod "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" (UID: "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe"). InnerVolumeSpecName "kube-api-access-h6zrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:24.121513 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.121490 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-podinfo" pod "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" (UID: "53e745f6-3ef0-4391-b1ec-2e0feb40ddfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:56:24.137261 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.137235 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp"] Oct 30 09:56:24.142780 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.142751 5938 scope.go:122] "RemoveContainer" containerID="c36ecb0d6a73b3279f6399132375799c3fbbd2e13178ee037eb117550b310582" Oct 30 09:56:24.143103 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:56:24.143082 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36ecb0d6a73b3279f6399132375799c3fbbd2e13178ee037eb117550b310582\": container with ID starting with c36ecb0d6a73b3279f6399132375799c3fbbd2e13178ee037eb117550b310582 not found: ID does not exist" containerID="c36ecb0d6a73b3279f6399132375799c3fbbd2e13178ee037eb117550b310582" Oct 30 09:56:24.143180 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.143108 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36ecb0d6a73b3279f6399132375799c3fbbd2e13178ee037eb117550b310582"} err="failed to get container status \"c36ecb0d6a73b3279f6399132375799c3fbbd2e13178ee037eb117550b310582\": rpc error: code = NotFound desc = could not find container \"c36ecb0d6a73b3279f6399132375799c3fbbd2e13178ee037eb117550b310582\": container with ID starting with c36ecb0d6a73b3279f6399132375799c3fbbd2e13178ee037eb117550b310582 not found: ID does not exist" Oct 30 09:56:24.143180 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.143126 5938 scope.go:122] "RemoveContainer" containerID="ec34ecee1d481d600b1cd060d276523bc3f8cf8e1526f700e85fa150bf40619d" Oct 30 09:56:24.143411 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:56:24.143393 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec34ecee1d481d600b1cd060d276523bc3f8cf8e1526f700e85fa150bf40619d\": container with ID starting with ec34ecee1d481d600b1cd060d276523bc3f8cf8e1526f700e85fa150bf40619d not found: ID does not exist" containerID="ec34ecee1d481d600b1cd060d276523bc3f8cf8e1526f700e85fa150bf40619d" Oct 30 09:56:24.143459 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.143421 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec34ecee1d481d600b1cd060d276523bc3f8cf8e1526f700e85fa150bf40619d"} err="failed to get container status \"ec34ecee1d481d600b1cd060d276523bc3f8cf8e1526f700e85fa150bf40619d\": rpc error: code = NotFound desc = could not find container \"ec34ecee1d481d600b1cd060d276523bc3f8cf8e1526f700e85fa150bf40619d\": container with ID starting with ec34ecee1d481d600b1cd060d276523bc3f8cf8e1526f700e85fa150bf40619d not found: ID does not exist" Oct 30 09:56:24.219636 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.219564 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.219636 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.219611 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.219636 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.219622 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6zrv\" (UniqueName: \"kubernetes.io/projected/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-kube-api-access-h6zrv\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.219636 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.219631 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.449056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.449026 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr"] Oct 30 09:56:24.462152 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.462124 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-gbqqr"] Oct 30 09:56:24.685449 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.685415 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v"] Oct 30 09:56:24.685738 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:24.685709 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" podUID="04620105-1d5a-40b4-bfc3-3531870174cc" containerName="maas-api" containerID="cri-o://bee4fc18f0befe6c4cefa2902878287fad3d4c5b360c35e91b8981184ecfd6c3" gracePeriod=30 Oct 30 09:56:25.035639 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.034491 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:56:25.160618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.157387 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04620105-1d5a-40b4-bfc3-3531870174cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04620105-1d5a-40b4-bfc3-3531870174cc-maas-api-tls\") pod \"04620105-1d5a-40b4-bfc3-3531870174cc\" (UID: \"04620105-1d5a-40b4-bfc3-3531870174cc\") " Oct 30 09:56:25.160618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.157571 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04620105-1d5a-40b4-bfc3-3531870174cc-kube-api-access-59xnp\" (UniqueName: \"kubernetes.io/projected/04620105-1d5a-40b4-bfc3-3531870174cc-kube-api-access-59xnp\") pod \"04620105-1d5a-40b4-bfc3-3531870174cc\" (UID: \"04620105-1d5a-40b4-bfc3-3531870174cc\") " Oct 30 09:56:25.160618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.157632 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04620105-1d5a-40b4-bfc3-3531870174cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04620105-1d5a-40b4-bfc3-3531870174cc-metrics-certs\") pod \"04620105-1d5a-40b4-bfc3-3531870174cc\" (UID: \"04620105-1d5a-40b4-bfc3-3531870174cc\") " Oct 30 09:56:25.160618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.157737 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04620105-1d5a-40b4-bfc3-3531870174cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04620105-1d5a-40b4-bfc3-3531870174cc-openshift-service-ca\") pod \"04620105-1d5a-40b4-bfc3-3531870174cc\" (UID: \"04620105-1d5a-40b4-bfc3-3531870174cc\") " Oct 30 09:56:25.160618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.160403 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04620105-1d5a-40b4-bfc3-3531870174cc-openshift-service-ca" pod "04620105-1d5a-40b4-bfc3-3531870174cc" (UID: "04620105-1d5a-40b4-bfc3-3531870174cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:25.174614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.173330 5938 generic.go:356] "Generic (PLEG): container finished" podID="04620105-1d5a-40b4-bfc3-3531870174cc" containerID="bee4fc18f0befe6c4cefa2902878287fad3d4c5b360c35e91b8981184ecfd6c3" exitCode=0 Oct 30 09:56:25.174614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.173504 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" event={"ID":"04620105-1d5a-40b4-bfc3-3531870174cc","Type":"ContainerDied","Data":"bee4fc18f0befe6c4cefa2902878287fad3d4c5b360c35e91b8981184ecfd6c3"} Oct 30 09:56:25.174614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.173556 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" event={"ID":"04620105-1d5a-40b4-bfc3-3531870174cc","Type":"ContainerDied","Data":"5fa53d4adc9d8302db0d50488ebac81ffac1df6c729ed3c5b6a5449c855c7fcb"} Oct 30 09:56:25.174614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.173608 5938 scope.go:122] "RemoveContainer" containerID="bee4fc18f0befe6c4cefa2902878287fad3d4c5b360c35e91b8981184ecfd6c3" Oct 30 09:56:25.174614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.173730 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v" Oct 30 09:56:25.176719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.176681 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04620105-1d5a-40b4-bfc3-3531870174cc-kube-api-access-59xnp" pod "04620105-1d5a-40b4-bfc3-3531870174cc" (UID: "04620105-1d5a-40b4-bfc3-3531870174cc"). InnerVolumeSpecName "kube-api-access-59xnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:25.182022 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.181994 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04620105-1d5a-40b4-bfc3-3531870174cc-metrics-certs" pod "04620105-1d5a-40b4-bfc3-3531870174cc" (UID: "04620105-1d5a-40b4-bfc3-3531870174cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:25.183846 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.183736 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04620105-1d5a-40b4-bfc3-3531870174cc-maas-api-tls" pod "04620105-1d5a-40b4-bfc3-3531870174cc" (UID: "04620105-1d5a-40b4-bfc3-3531870174cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:25.191092 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.191074 5938 scope.go:122] "RemoveContainer" containerID="bee4fc18f0befe6c4cefa2902878287fad3d4c5b360c35e91b8981184ecfd6c3" Oct 30 09:56:25.191651 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:56:25.191629 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bee4fc18f0befe6c4cefa2902878287fad3d4c5b360c35e91b8981184ecfd6c3\": container with ID starting with bee4fc18f0befe6c4cefa2902878287fad3d4c5b360c35e91b8981184ecfd6c3 not found: ID does not exist" containerID="bee4fc18f0befe6c4cefa2902878287fad3d4c5b360c35e91b8981184ecfd6c3" Oct 30 09:56:25.191795 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.191769 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bee4fc18f0befe6c4cefa2902878287fad3d4c5b360c35e91b8981184ecfd6c3"} err="failed to get container status \"bee4fc18f0befe6c4cefa2902878287fad3d4c5b360c35e91b8981184ecfd6c3\": rpc error: code = NotFound desc = could not find container \"bee4fc18f0befe6c4cefa2902878287fad3d4c5b360c35e91b8981184ecfd6c3\": container with ID starting with bee4fc18f0befe6c4cefa2902878287fad3d4c5b360c35e91b8981184ecfd6c3 not found: ID does not exist" Oct 30 09:56:25.259144 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.259065 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59xnp\" (UniqueName: \"kubernetes.io/projected/04620105-1d5a-40b4-bfc3-3531870174cc-kube-api-access-59xnp\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:25.259144 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.259093 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04620105-1d5a-40b4-bfc3-3531870174cc-metrics-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:25.259144 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.259107 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04620105-1d5a-40b4-bfc3-3531870174cc-openshift-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:25.259144 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.259120 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04620105-1d5a-40b4-bfc3-3531870174cc-maas-api-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:25.508078 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.508049 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5"] Oct 30 09:56:25.514629 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.514556 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-304459-95pr5"] Oct 30 09:56:25.521813 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.521788 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v"] Oct 30 09:56:25.524831 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:25.524809 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-304459-6cb446f967-jvs4v"] Oct 30 09:56:26.259914 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:26.259880 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04620105-1d5a-40b4-bfc3-3531870174cc" path="/var/lib/kubelet/pods/04620105-1d5a-40b4-bfc3-3531870174cc/volumes" Oct 30 09:56:26.260752 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:26.260728 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" path="/var/lib/kubelet/pods/53e745f6-3ef0-4391-b1ec-2e0feb40ddfe/volumes" Oct 30 09:56:26.261243 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:26.261224 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d409189c-887a-4f67-a7d4-acf3ef0273c8" path="/var/lib/kubelet/pods/d409189c-887a-4f67-a7d4-acf3ef0273c8/volumes" Oct 30 09:56:30.464521 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:30.464486 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5"] Oct 30 09:56:32.140220 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.140183 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx"] Oct 30 09:56:32.140888 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.140424 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" podUID="aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" containerName="istio-proxy" containerID="cri-o://8d7ae38d765acac1fc64316d785e15986dbc83a464f859090b179d1091ed2c2c" gracePeriod=30 Oct 30 09:56:32.732626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.732576 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t"] Oct 30 09:56:32.733173 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.733158 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9c62c859-8cd1-478e-8f0c-d67506b620cd" containerName="revoke-keys" Oct 30 09:56:32.733226 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.733175 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c62c859-8cd1-478e-8f0c-d67506b620cd" containerName="revoke-keys" Oct 30 09:56:32.733226 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.733205 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2f722682-e624-46fe-81f0-c630c2718a54" containerName="revoke-keys" Oct 30 09:56:32.733226 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.733210 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f722682-e624-46fe-81f0-c630c2718a54" containerName="revoke-keys" Oct 30 09:56:32.733226 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.733218 5938 cpu_manager.go:414] "containerMap: removing container" podUID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" containerName="istio-proxy" Oct 30 09:56:32.733226 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.733224 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" containerName="istio-proxy" Oct 30 09:56:32.733389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.733238 5938 cpu_manager.go:414] "containerMap: removing container" podUID="04620105-1d5a-40b4-bfc3-3531870174cc" containerName="maas-api" Oct 30 09:56:32.733389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.733243 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04620105-1d5a-40b4-bfc3-3531870174cc" containerName="maas-api" Oct 30 09:56:32.733389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.733354 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" containerName="istio-proxy" Oct 30 09:56:32.733389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.733364 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="04620105-1d5a-40b4-bfc3-3531870174cc" containerName="maas-api" Oct 30 09:56:32.733389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.733376 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" containerName="istio-proxy" Oct 30 09:56:32.733389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.733382 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c62c859-8cd1-478e-8f0c-d67506b620cd" containerName="revoke-keys" Oct 30 09:56:32.733389 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.733391 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f722682-e624-46fe-81f0-c630c2718a54" containerName="revoke-keys" Oct 30 09:56:32.738631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.738614 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" Oct 30 09:56:32.754288 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.754264 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:56:32.754552 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.754437 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:56:32.755078 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.755056 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t"] Oct 30 09:56:32.852493 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.852470 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kgrh\" (UniqueName: \"kubernetes.io/projected/a5dc127e-23e5-4fac-94f7-6a33c7998515-kube-api-access-5kgrh\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t\" (UID: \"a5dc127e-23e5-4fac-94f7-6a33c7998515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" Oct 30 09:56:32.852688 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.852668 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5dc127e-23e5-4fac-94f7-6a33c7998515-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t\" (UID: \"a5dc127e-23e5-4fac-94f7-6a33c7998515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" Oct 30 09:56:32.852747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.852731 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5dc127e-23e5-4fac-94f7-6a33c7998515-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t\" (UID: \"a5dc127e-23e5-4fac-94f7-6a33c7998515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" Oct 30 09:56:32.953632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.953607 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kgrh\" (UniqueName: \"kubernetes.io/projected/a5dc127e-23e5-4fac-94f7-6a33c7998515-kube-api-access-5kgrh\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t\" (UID: \"a5dc127e-23e5-4fac-94f7-6a33c7998515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" Oct 30 09:56:32.953763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.953747 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5dc127e-23e5-4fac-94f7-6a33c7998515-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t\" (UID: \"a5dc127e-23e5-4fac-94f7-6a33c7998515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" Oct 30 09:56:32.953824 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.953807 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5dc127e-23e5-4fac-94f7-6a33c7998515-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t\" (UID: \"a5dc127e-23e5-4fac-94f7-6a33c7998515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" Oct 30 09:56:32.954359 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.954336 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5dc127e-23e5-4fac-94f7-6a33c7998515-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t\" (UID: \"a5dc127e-23e5-4fac-94f7-6a33c7998515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" Oct 30 09:56:32.956307 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.956284 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5dc127e-23e5-4fac-94f7-6a33c7998515-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t\" (UID: \"a5dc127e-23e5-4fac-94f7-6a33c7998515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" Oct 30 09:56:32.973856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:32.973829 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kgrh\" (UniqueName: \"kubernetes.io/projected/a5dc127e-23e5-4fac-94f7-6a33c7998515-kube-api-access-5kgrh\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t\" (UID: \"a5dc127e-23e5-4fac-94f7-6a33c7998515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" Oct 30 09:56:33.052307 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:33.052254 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" Oct 30 09:56:33.359693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:33.359660 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t"] Oct 30 09:56:33.360628 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:56:33.360566 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5dc127e_23e5_4fac_94f7_6a33c7998515.slice/crio-25540a0eb591e00272d6ef80fa112a366b6cb04912493c0d5f2d3e669021aa9e WatchSource:0}: Error finding container 25540a0eb591e00272d6ef80fa112a366b6cb04912493c0d5f2d3e669021aa9e: Status 404 returned error can't find the container with id 25540a0eb591e00272d6ef80fa112a366b6cb04912493c0d5f2d3e669021aa9e Oct 30 09:56:34.268094 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:34.268058 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" event={"ID":"a5dc127e-23e5-4fac-94f7-6a33c7998515","Type":"ContainerStarted","Data":"f411c1e0dfc029ca4dc30bad38d0000aca0f1974687e26886c70e1b6b1a7ee62"} Oct 30 09:56:34.268263 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:34.268121 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" event={"ID":"a5dc127e-23e5-4fac-94f7-6a33c7998515","Type":"ContainerStarted","Data":"25540a0eb591e00272d6ef80fa112a366b6cb04912493c0d5f2d3e669021aa9e"} Oct 30 09:56:34.309398 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:34.309356 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" podStartSLOduration=2.309339264 podStartE2EDuration="2.309339264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:56:34.307464271 +0000 UTC m=+3712.170170092" watchObservedRunningTime="2026-10-30 09:56:34.309339264 +0000 UTC m=+3712.172045096" Oct 30 09:56:35.276967 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:35.276930 5938 generic.go:356] "Generic (PLEG): container finished" podID="a5dc127e-23e5-4fac-94f7-6a33c7998515" containerID="f411c1e0dfc029ca4dc30bad38d0000aca0f1974687e26886c70e1b6b1a7ee62" exitCode=0 Oct 30 09:56:35.277434 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:35.277010 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" event={"ID":"a5dc127e-23e5-4fac-94f7-6a33c7998515","Type":"ContainerDied","Data":"f411c1e0dfc029ca4dc30bad38d0000aca0f1974687e26886c70e1b6b1a7ee62"} Oct 30 09:56:35.465822 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:35.465782 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" podUID="c82833df-fd04-431e-8bc2-693a47dfb44e" containerName="payload-processing" containerID="cri-o://6c4589611a6b80b1657e1d7efa51238d5187c448e4a4872336f6709465fc19d7" gracePeriod=25 Oct 30 09:56:36.445645 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:36.445621 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" Oct 30 09:56:36.494773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:36.494744 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5dc127e-23e5-4fac-94f7-6a33c7998515-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5dc127e-23e5-4fac-94f7-6a33c7998515-maas-api-xks-ca\") pod \"a5dc127e-23e5-4fac-94f7-6a33c7998515\" (UID: \"a5dc127e-23e5-4fac-94f7-6a33c7998515\") " Oct 30 09:56:36.494933 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:36.494844 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5dc127e-23e5-4fac-94f7-6a33c7998515-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5dc127e-23e5-4fac-94f7-6a33c7998515-maas-api-openshift-ca\") pod \"a5dc127e-23e5-4fac-94f7-6a33c7998515\" (UID: \"a5dc127e-23e5-4fac-94f7-6a33c7998515\") " Oct 30 09:56:36.494933 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:36.494922 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5dc127e-23e5-4fac-94f7-6a33c7998515-kube-api-access-5kgrh\" (UniqueName: \"kubernetes.io/projected/a5dc127e-23e5-4fac-94f7-6a33c7998515-kube-api-access-5kgrh\") pod \"a5dc127e-23e5-4fac-94f7-6a33c7998515\" (UID: \"a5dc127e-23e5-4fac-94f7-6a33c7998515\") " Oct 30 09:56:36.495241 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:36.495214 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5dc127e-23e5-4fac-94f7-6a33c7998515-maas-api-openshift-ca" pod "a5dc127e-23e5-4fac-94f7-6a33c7998515" (UID: "a5dc127e-23e5-4fac-94f7-6a33c7998515"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:36.495484 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:36.495463 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5dc127e-23e5-4fac-94f7-6a33c7998515-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:36.497236 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:36.497215 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5dc127e-23e5-4fac-94f7-6a33c7998515-kube-api-access-5kgrh" pod "a5dc127e-23e5-4fac-94f7-6a33c7998515" (UID: "a5dc127e-23e5-4fac-94f7-6a33c7998515"). InnerVolumeSpecName "kube-api-access-5kgrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:36.497471 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:36.497458 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5dc127e-23e5-4fac-94f7-6a33c7998515-maas-api-xks-ca" pod "a5dc127e-23e5-4fac-94f7-6a33c7998515" (UID: "a5dc127e-23e5-4fac-94f7-6a33c7998515"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:36.596442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:36.596376 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5dc127e-23e5-4fac-94f7-6a33c7998515-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:36.596442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:36.596406 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kgrh\" (UniqueName: \"kubernetes.io/projected/a5dc127e-23e5-4fac-94f7-6a33c7998515-kube-api-access-5kgrh\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.294906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.294871 5938 generic.go:356] "Generic (PLEG): container finished" podID="aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" containerID="8d7ae38d765acac1fc64316d785e15986dbc83a464f859090b179d1091ed2c2c" exitCode=0 Oct 30 09:56:37.295154 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.294901 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" event={"ID":"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f","Type":"ContainerDied","Data":"8d7ae38d765acac1fc64316d785e15986dbc83a464f859090b179d1091ed2c2c"} Oct 30 09:56:37.296518 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.296500 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" Oct 30 09:56:37.296685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.296532 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t" event={"ID":"a5dc127e-23e5-4fac-94f7-6a33c7998515","Type":"ContainerDied","Data":"25540a0eb591e00272d6ef80fa112a366b6cb04912493c0d5f2d3e669021aa9e"} Oct 30 09:56:37.296685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.296599 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25540a0eb591e00272d6ef80fa112a366b6cb04912493c0d5f2d3e669021aa9e" Oct 30 09:56:37.393882 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.393857 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:56:37.507266 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.507237 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-kube-api-access-fj9vc\" (UniqueName: \"kubernetes.io/projected/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-kube-api-access-fj9vc\") pod \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " Oct 30 09:56:37.507752 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.507279 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-envoy\") pod \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " Oct 30 09:56:37.507752 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.507321 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-credential-socket\") pod \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " Oct 30 09:56:37.507752 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.507370 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-podinfo\") pod \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " Oct 30 09:56:37.507752 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.507462 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-data\") pod \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " Oct 30 09:56:37.507752 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.507528 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-workload-socket\") pod \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " Oct 30 09:56:37.507752 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.507567 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-token\" (UniqueName: \"kubernetes.io/projected/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-token\") pod \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " Oct 30 09:56:37.507752 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.507682 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-workload-certs\") pod \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " Oct 30 09:56:37.507752 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.507702 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-credential-socket" pod "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" (UID: "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:37.508432 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.507777 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istiod-ca-cert\") pod \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\" (UID: \"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f\") " Oct 30 09:56:37.508432 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.507814 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-workload-socket" pod "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" (UID: "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:37.508432 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.507881 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-data" pod "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" (UID: "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:37.508432 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.508033 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-workload-certs" pod "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" (UID: "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:37.508707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.508473 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istiod-ca-cert" pod "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" (UID: "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:37.508707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.508681 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.508707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.508702 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.508886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.508716 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.508886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.508732 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.508886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.508744 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.509855 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.509823 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-envoy" pod "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" (UID: "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:37.509973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.509909 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-token" pod "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" (UID: "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:37.509973 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.509953 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-kube-api-access-fj9vc" pod "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" (UID: "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f"). InnerVolumeSpecName "kube-api-access-fj9vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:37.510056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.509978 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-podinfo" pod "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" (UID: "aa3ac3bb-d25c-44cf-b6f8-283b74bc738f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:56:37.610371 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.610343 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.610371 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.610372 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj9vc\" (UniqueName: \"kubernetes.io/projected/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-kube-api-access-fj9vc\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.610544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.610384 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.610544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:37.610396 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:38.226090 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.226055 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk"] Oct 30 09:56:38.226780 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.226761 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a5dc127e-23e5-4fac-94f7-6a33c7998515" containerName="revoke-keys" Oct 30 09:56:38.226780 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.226780 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5dc127e-23e5-4fac-94f7-6a33c7998515" containerName="revoke-keys" Oct 30 09:56:38.226917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.226825 5938 cpu_manager.go:414] "containerMap: removing container" podUID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" containerName="istio-proxy" Oct 30 09:56:38.226917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.226833 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e745f6-3ef0-4391-b1ec-2e0feb40ddfe" containerName="istio-proxy" Oct 30 09:56:38.226917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.226843 5938 cpu_manager.go:414] "containerMap: removing container" podUID="aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" containerName="istio-proxy" Oct 30 09:56:38.226917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.226850 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" containerName="istio-proxy" Oct 30 09:56:38.227043 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.226965 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" containerName="istio-proxy" Oct 30 09:56:38.227043 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.226976 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5dc127e-23e5-4fac-94f7-6a33c7998515" containerName="revoke-keys" Oct 30 09:56:38.230915 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.230896 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" Oct 30 09:56:38.249165 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.249131 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk"] Oct 30 09:56:38.257224 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.257171 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:56:38.257365 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.257294 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:56:38.306312 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.306287 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" Oct 30 09:56:38.306440 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.306284 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx" event={"ID":"aa3ac3bb-d25c-44cf-b6f8-283b74bc738f","Type":"ContainerDied","Data":"50e18671e1b38bfadbb8ef1749d6f9c582d30561bc934a1afb37b225f3e8a2ef"} Oct 30 09:56:38.306440 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.306409 5938 scope.go:122] "RemoveContainer" containerID="8d7ae38d765acac1fc64316d785e15986dbc83a464f859090b179d1091ed2c2c" Oct 30 09:56:38.319642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.319370 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40a8f23-0eaa-4785-940d-595544bbf8df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97c9c63e-hpjdk\" (UID: \"e40a8f23-0eaa-4785-940d-595544bbf8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" Oct 30 09:56:38.319774 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.319691 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40a8f23-0eaa-4785-940d-595544bbf8df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97c9c63e-hpjdk\" (UID: \"e40a8f23-0eaa-4785-940d-595544bbf8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" Oct 30 09:56:38.320066 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.320039 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bd52\" (UniqueName: \"kubernetes.io/projected/e40a8f23-0eaa-4785-940d-595544bbf8df-kube-api-access-9bd52\") pod \"maas-api-revoke-sub-97c9c63e-hpjdk\" (UID: \"e40a8f23-0eaa-4785-940d-595544bbf8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" Oct 30 09:56:38.364892 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.364858 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx"] Oct 30 09:56:38.375241 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.375220 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-zn2gx"] Oct 30 09:56:38.421819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.421784 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40a8f23-0eaa-4785-940d-595544bbf8df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97c9c63e-hpjdk\" (UID: \"e40a8f23-0eaa-4785-940d-595544bbf8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" Oct 30 09:56:38.421953 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.421859 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40a8f23-0eaa-4785-940d-595544bbf8df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97c9c63e-hpjdk\" (UID: \"e40a8f23-0eaa-4785-940d-595544bbf8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" Oct 30 09:56:38.422064 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.422041 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bd52\" (UniqueName: \"kubernetes.io/projected/e40a8f23-0eaa-4785-940d-595544bbf8df-kube-api-access-9bd52\") pod \"maas-api-revoke-sub-97c9c63e-hpjdk\" (UID: \"e40a8f23-0eaa-4785-940d-595544bbf8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" Oct 30 09:56:38.422815 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.422781 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40a8f23-0eaa-4785-940d-595544bbf8df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97c9c63e-hpjdk\" (UID: \"e40a8f23-0eaa-4785-940d-595544bbf8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" Oct 30 09:56:38.424541 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.424515 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40a8f23-0eaa-4785-940d-595544bbf8df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97c9c63e-hpjdk\" (UID: \"e40a8f23-0eaa-4785-940d-595544bbf8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" Oct 30 09:56:38.443798 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.443771 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bd52\" (UniqueName: \"kubernetes.io/projected/e40a8f23-0eaa-4785-940d-595544bbf8df-kube-api-access-9bd52\") pod \"maas-api-revoke-sub-97c9c63e-hpjdk\" (UID: \"e40a8f23-0eaa-4785-940d-595544bbf8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" Oct 30 09:56:38.546559 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.546470 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" Oct 30 09:56:38.860923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:38.860895 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk"] Oct 30 09:56:38.862422 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:56:38.862394 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40a8f23_0eaa_4785_940d_595544bbf8df.slice/crio-696719307dcca6f9e9d86b9d698b84b6035a83b4b68733022625fa683acca7e6 WatchSource:0}: Error finding container 696719307dcca6f9e9d86b9d698b84b6035a83b4b68733022625fa683acca7e6: Status 404 returned error can't find the container with id 696719307dcca6f9e9d86b9d698b84b6035a83b4b68733022625fa683acca7e6 Oct 30 09:56:39.159399 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.159369 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" containerID="cri-o://d4ca7ff0509f3a5575c9dd9d60c5702ca14d8f2ee08c5ca6e52c9610db592b09" gracePeriod=45 Oct 30 09:56:39.318114 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.318074 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" event={"ID":"e40a8f23-0eaa-4785-940d-595544bbf8df","Type":"ContainerStarted","Data":"d62422f910c2329ff74db8ba6903f920ff99dd2027724028d5bffb4573993fb0"} Oct 30 09:56:39.318318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.318145 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" event={"ID":"e40a8f23-0eaa-4785-940d-595544bbf8df","Type":"ContainerStarted","Data":"696719307dcca6f9e9d86b9d698b84b6035a83b4b68733022625fa683acca7e6"} Oct 30 09:56:39.319855 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.319830 5938 generic.go:356] "Generic (PLEG): container finished" podID="0aa496ad-244c-437e-aeba-630c32464259" containerID="d4ca7ff0509f3a5575c9dd9d60c5702ca14d8f2ee08c5ca6e52c9610db592b09" exitCode=0 Oct 30 09:56:39.320013 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.319922 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" event={"ID":"0aa496ad-244c-437e-aeba-630c32464259","Type":"ContainerDied","Data":"d4ca7ff0509f3a5575c9dd9d60c5702ca14d8f2ee08c5ca6e52c9610db592b09"} Oct 30 09:56:39.372451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.372387 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" podStartSLOduration=1.3723642090000001 podStartE2EDuration="1.372364209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:56:39.357015111 +0000 UTC m=+3717.219720932" watchObservedRunningTime="2026-10-30 09:56:39.372364209 +0000 UTC m=+3717.235070031" Oct 30 09:56:39.399268 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.399243 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:56:39.537457 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.537427 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-tmp-dir\") pod \"0aa496ad-244c-437e-aeba-630c32464259\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " Oct 30 09:56:39.537656 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.537508 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa496ad-244c-437e-aeba-630c32464259-kube-api-access-954gl\" (UniqueName: \"kubernetes.io/projected/0aa496ad-244c-437e-aeba-630c32464259-kube-api-access-954gl\") pod \"0aa496ad-244c-437e-aeba-630c32464259\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " Oct 30 09:56:39.537656 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.537550 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-home\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-home\") pod \"0aa496ad-244c-437e-aeba-630c32464259\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " Oct 30 09:56:39.537790 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.537700 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aa496ad-244c-437e-aeba-630c32464259-tls-certs\" (UniqueName: \"kubernetes.io/secret/0aa496ad-244c-437e-aeba-630c32464259-tls-certs\") pod \"0aa496ad-244c-437e-aeba-630c32464259\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " Oct 30 09:56:39.537790 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.537700 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-tmp-dir" pod "0aa496ad-244c-437e-aeba-630c32464259" (UID: "0aa496ad-244c-437e-aeba-630c32464259"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:39.537897 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.537794 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-dshm\") pod \"0aa496ad-244c-437e-aeba-630c32464259\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " Oct 30 09:56:39.537952 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.537901 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-model-cache\") pod \"0aa496ad-244c-437e-aeba-630c32464259\" (UID: \"0aa496ad-244c-437e-aeba-630c32464259\") " Oct 30 09:56:39.537952 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.537938 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-home" pod "0aa496ad-244c-437e-aeba-630c32464259" (UID: "0aa496ad-244c-437e-aeba-630c32464259"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:39.538227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.538203 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-model-cache" pod "0aa496ad-244c-437e-aeba-630c32464259" (UID: "0aa496ad-244c-437e-aeba-630c32464259"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:39.538488 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.538470 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.538569 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.538493 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.538569 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.538510 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.540035 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.540008 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aa496ad-244c-437e-aeba-630c32464259-tls-certs" pod "0aa496ad-244c-437e-aeba-630c32464259" (UID: "0aa496ad-244c-437e-aeba-630c32464259"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:39.540158 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.540106 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa496ad-244c-437e-aeba-630c32464259-kube-api-access-954gl" pod "0aa496ad-244c-437e-aeba-630c32464259" (UID: "0aa496ad-244c-437e-aeba-630c32464259"). InnerVolumeSpecName "kube-api-access-954gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:39.540267 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.540237 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-dshm" pod "0aa496ad-244c-437e-aeba-630c32464259" (UID: "0aa496ad-244c-437e-aeba-630c32464259"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:39.639735 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.639707 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-954gl\" (UniqueName: \"kubernetes.io/projected/0aa496ad-244c-437e-aeba-630c32464259-kube-api-access-954gl\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.639735 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.639732 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aa496ad-244c-437e-aeba-630c32464259-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.639735 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:39.639743 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aa496ad-244c-437e-aeba-630c32464259-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:40.257971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:40.257937 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa3ac3bb-d25c-44cf-b6f8-283b74bc738f" path="/var/lib/kubelet/pods/aa3ac3bb-d25c-44cf-b6f8-283b74bc738f/volumes" Oct 30 09:56:40.330443 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:40.330415 5938 generic.go:356] "Generic (PLEG): container finished" podID="e40a8f23-0eaa-4785-940d-595544bbf8df" containerID="d62422f910c2329ff74db8ba6903f920ff99dd2027724028d5bffb4573993fb0" exitCode=0 Oct 30 09:56:40.330602 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:40.330501 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" event={"ID":"e40a8f23-0eaa-4785-940d-595544bbf8df","Type":"ContainerDied","Data":"d62422f910c2329ff74db8ba6903f920ff99dd2027724028d5bffb4573993fb0"} Oct 30 09:56:40.332149 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:40.332120 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" Oct 30 09:56:40.332278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:40.332143 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp" event={"ID":"0aa496ad-244c-437e-aeba-630c32464259","Type":"ContainerDied","Data":"34040528e097241820cccaf1dd92dbf370c1757a595257a046e2977d4932343c"} Oct 30 09:56:40.332278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:40.332188 5938 scope.go:122] "RemoveContainer" containerID="d4ca7ff0509f3a5575c9dd9d60c5702ca14d8f2ee08c5ca6e52c9610db592b09" Oct 30 09:56:40.411693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:40.411655 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp"] Oct 30 09:56:40.414271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:40.414247 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/premium-w1-c5447d-kserve-6cc8fbc75b-r9jgp"] Oct 30 09:56:41.491944 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:41.491914 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" Oct 30 09:56:41.562720 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:41.562691 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e40a8f23-0eaa-4785-940d-595544bbf8df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40a8f23-0eaa-4785-940d-595544bbf8df-maas-api-xks-ca\") pod \"e40a8f23-0eaa-4785-940d-595544bbf8df\" (UID: \"e40a8f23-0eaa-4785-940d-595544bbf8df\") " Oct 30 09:56:41.562887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:41.562821 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40a8f23-0eaa-4785-940d-595544bbf8df-kube-api-access-9bd52\" (UniqueName: \"kubernetes.io/projected/e40a8f23-0eaa-4785-940d-595544bbf8df-kube-api-access-9bd52\") pod \"e40a8f23-0eaa-4785-940d-595544bbf8df\" (UID: \"e40a8f23-0eaa-4785-940d-595544bbf8df\") " Oct 30 09:56:41.562988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:41.562969 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e40a8f23-0eaa-4785-940d-595544bbf8df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40a8f23-0eaa-4785-940d-595544bbf8df-maas-api-openshift-ca\") pod \"e40a8f23-0eaa-4785-940d-595544bbf8df\" (UID: \"e40a8f23-0eaa-4785-940d-595544bbf8df\") " Oct 30 09:56:41.563334 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:41.563294 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e40a8f23-0eaa-4785-940d-595544bbf8df-maas-api-openshift-ca" pod "e40a8f23-0eaa-4785-940d-595544bbf8df" (UID: "e40a8f23-0eaa-4785-940d-595544bbf8df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:41.563536 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:41.563518 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40a8f23-0eaa-4785-940d-595544bbf8df-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:41.565499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:41.565477 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e40a8f23-0eaa-4785-940d-595544bbf8df-maas-api-xks-ca" pod "e40a8f23-0eaa-4785-940d-595544bbf8df" (UID: "e40a8f23-0eaa-4785-940d-595544bbf8df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:41.565650 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:41.565494 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40a8f23-0eaa-4785-940d-595544bbf8df-kube-api-access-9bd52" pod "e40a8f23-0eaa-4785-940d-595544bbf8df" (UID: "e40a8f23-0eaa-4785-940d-595544bbf8df"). InnerVolumeSpecName "kube-api-access-9bd52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:41.665766 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:41.665723 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40a8f23-0eaa-4785-940d-595544bbf8df-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:41.665766 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:41.665765 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bd52\" (UniqueName: \"kubernetes.io/projected/e40a8f23-0eaa-4785-940d-595544bbf8df-kube-api-access-9bd52\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:42.260341 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:42.260306 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aa496ad-244c-437e-aeba-630c32464259" path="/var/lib/kubelet/pods/0aa496ad-244c-437e-aeba-630c32464259/volumes" Oct 30 09:56:42.348680 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:42.348644 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" Oct 30 09:56:42.348828 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:42.348696 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk" event={"ID":"e40a8f23-0eaa-4785-940d-595544bbf8df","Type":"ContainerDied","Data":"696719307dcca6f9e9d86b9d698b84b6035a83b4b68733022625fa683acca7e6"} Oct 30 09:56:42.348828 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:42.348740 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="696719307dcca6f9e9d86b9d698b84b6035a83b4b68733022625fa683acca7e6" Oct 30 09:56:44.379763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:44.379718 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" podUID="c82833df-fd04-431e-8bc2-693a47dfb44e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:56:45.808515 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:56:45.808479 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:56:48.258531 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:48.258498 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h"] Oct 30 09:56:48.271544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:48.271513 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0dc148-kng4h"] Oct 30 09:56:49.656669 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.656639 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn"] Oct 30 09:56:49.657244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.657230 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" Oct 30 09:56:49.657294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.657245 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" Oct 30 09:56:49.657294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.657274 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e40a8f23-0eaa-4785-940d-595544bbf8df" containerName="revoke-keys" Oct 30 09:56:49.657294 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.657278 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e40a8f23-0eaa-4785-940d-595544bbf8df" containerName="revoke-keys" Oct 30 09:56:49.657390 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.657361 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e40a8f23-0eaa-4785-940d-595544bbf8df" containerName="revoke-keys" Oct 30 09:56:49.657390 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.657377 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa496ad-244c-437e-aeba-630c32464259" containerName="main" Oct 30 09:56:49.660984 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.660958 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn" Oct 30 09:56:49.671217 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.671194 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6892707d-6965-4b58-8e66-283364988a03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c2c1da1-6z5nn\" (UID: \"6892707d-6965-4b58-8e66-283364988a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn" Oct 30 09:56:49.671395 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.671371 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6892707d-6965-4b58-8e66-283364988a03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c2c1da1-6z5nn\" (UID: \"6892707d-6965-4b58-8e66-283364988a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn" Oct 30 09:56:49.671564 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.671542 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6r2v\" (UniqueName: \"kubernetes.io/projected/6892707d-6965-4b58-8e66-283364988a03-kube-api-access-f6r2v\") pod \"maas-api-revoke-sub-8c2c1da1-6z5nn\" (UID: \"6892707d-6965-4b58-8e66-283364988a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn" Oct 30 09:56:49.692280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.692249 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:56:49.692417 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.692311 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:56:49.695975 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.695950 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn"] Oct 30 09:56:49.772662 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.772635 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6r2v\" (UniqueName: \"kubernetes.io/projected/6892707d-6965-4b58-8e66-283364988a03-kube-api-access-f6r2v\") pod \"maas-api-revoke-sub-8c2c1da1-6z5nn\" (UID: \"6892707d-6965-4b58-8e66-283364988a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn" Oct 30 09:56:49.772828 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.772770 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6892707d-6965-4b58-8e66-283364988a03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c2c1da1-6z5nn\" (UID: \"6892707d-6965-4b58-8e66-283364988a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn" Oct 30 09:56:49.772899 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.772881 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6892707d-6965-4b58-8e66-283364988a03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c2c1da1-6z5nn\" (UID: \"6892707d-6965-4b58-8e66-283364988a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn" Oct 30 09:56:49.773423 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.773402 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6892707d-6965-4b58-8e66-283364988a03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c2c1da1-6z5nn\" (UID: \"6892707d-6965-4b58-8e66-283364988a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn" Oct 30 09:56:49.775273 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.775254 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6892707d-6965-4b58-8e66-283364988a03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c2c1da1-6z5nn\" (UID: \"6892707d-6965-4b58-8e66-283364988a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn" Oct 30 09:56:49.801007 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.800978 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6r2v\" (UniqueName: \"kubernetes.io/projected/6892707d-6965-4b58-8e66-283364988a03-kube-api-access-f6r2v\") pod \"maas-api-revoke-sub-8c2c1da1-6z5nn\" (UID: \"6892707d-6965-4b58-8e66-283364988a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn" Oct 30 09:56:49.978558 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:49.978489 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn" Oct 30 09:56:50.257329 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:50.257258 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f722682-e624-46fe-81f0-c630c2718a54" path="/var/lib/kubelet/pods/2f722682-e624-46fe-81f0-c630c2718a54/volumes" Oct 30 09:56:50.278331 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:50.278305 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn"] Oct 30 09:56:50.279565 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:56:50.279538 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6892707d_6965_4b58_8e66_283364988a03.slice/crio-6de71e25353a978aae65b72e5dfc323b76365a8c02356a1396e501c5fec43599 WatchSource:0}: Error finding container 6de71e25353a978aae65b72e5dfc323b76365a8c02356a1396e501c5fec43599: Status 404 returned error can't find the container with id 6de71e25353a978aae65b72e5dfc323b76365a8c02356a1396e501c5fec43599 Oct 30 09:56:50.410258 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:50.410219 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn" event={"ID":"6892707d-6965-4b58-8e66-283364988a03","Type":"ContainerStarted","Data":"6de71e25353a978aae65b72e5dfc323b76365a8c02356a1396e501c5fec43599"} Oct 30 09:56:51.419936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:51.419903 5938 generic.go:356] "Generic (PLEG): container finished" podID="6892707d-6965-4b58-8e66-283364988a03" containerID="2425490d28e197f8ba247dfd09015a5fb741b1d0aaf6efa492cd49ceb274f66b" exitCode=0 Oct 30 09:56:51.420392 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:51.419981 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn" event={"ID":"6892707d-6965-4b58-8e66-283364988a03","Type":"ContainerDied","Data":"2425490d28e197f8ba247dfd09015a5fb741b1d0aaf6efa492cd49ceb274f66b"} Oct 30 09:56:52.624027 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.623995 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn" Oct 30 09:56:52.715618 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.715523 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6892707d-6965-4b58-8e66-283364988a03-kube-api-access-f6r2v\" (UniqueName: \"kubernetes.io/projected/6892707d-6965-4b58-8e66-283364988a03-kube-api-access-f6r2v\") pod \"6892707d-6965-4b58-8e66-283364988a03\" (UID: \"6892707d-6965-4b58-8e66-283364988a03\") " Oct 30 09:56:52.715816 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.715750 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6892707d-6965-4b58-8e66-283364988a03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6892707d-6965-4b58-8e66-283364988a03-maas-api-openshift-ca\") pod \"6892707d-6965-4b58-8e66-283364988a03\" (UID: \"6892707d-6965-4b58-8e66-283364988a03\") " Oct 30 09:56:52.715897 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.715836 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6892707d-6965-4b58-8e66-283364988a03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6892707d-6965-4b58-8e66-283364988a03-maas-api-xks-ca\") pod \"6892707d-6965-4b58-8e66-283364988a03\" (UID: \"6892707d-6965-4b58-8e66-283364988a03\") " Oct 30 09:56:52.716168 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.716137 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6892707d-6965-4b58-8e66-283364988a03-maas-api-openshift-ca" pod "6892707d-6965-4b58-8e66-283364988a03" (UID: "6892707d-6965-4b58-8e66-283364988a03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:52.718052 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.718025 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6892707d-6965-4b58-8e66-283364988a03-kube-api-access-f6r2v" pod "6892707d-6965-4b58-8e66-283364988a03" (UID: "6892707d-6965-4b58-8e66-283364988a03"). InnerVolumeSpecName "kube-api-access-f6r2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:52.718370 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.718356 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6892707d-6965-4b58-8e66-283364988a03-maas-api-xks-ca" pod "6892707d-6965-4b58-8e66-283364988a03" (UID: "6892707d-6965-4b58-8e66-283364988a03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:52.818005 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.817982 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6892707d-6965-4b58-8e66-283364988a03-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:52.818005 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.818006 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6892707d-6965-4b58-8e66-283364988a03-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:52.818137 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.818018 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6r2v\" (UniqueName: \"kubernetes.io/projected/6892707d-6965-4b58-8e66-283364988a03-kube-api-access-f6r2v\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:52.839807 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.839769 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5"] Oct 30 09:56:52.840462 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.840442 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6892707d-6965-4b58-8e66-283364988a03" containerName="revoke-keys" Oct 30 09:56:52.840462 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.840460 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6892707d-6965-4b58-8e66-283364988a03" containerName="revoke-keys" Oct 30 09:56:52.840648 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.840580 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6892707d-6965-4b58-8e66-283364988a03" containerName="revoke-keys" Oct 30 09:56:52.844193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.844174 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5" Oct 30 09:56:52.860384 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.860362 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q"] Oct 30 09:56:52.865470 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.865454 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5"] Oct 30 09:56:52.865628 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.865537 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q" Oct 30 09:56:52.884013 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.883987 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q"] Oct 30 09:56:52.919562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.919537 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d68d8a-8f84-4419-8344-cfe9d883178e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7766142-lg8t5\" (UID: \"20d68d8a-8f84-4419-8344-cfe9d883178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5" Oct 30 09:56:52.919706 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.919569 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22161d0f-3e3c-44db-acfb-81ac4f26a55c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a710ab-hqk2q\" (UID: \"22161d0f-3e3c-44db-acfb-81ac4f26a55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q" Oct 30 09:56:52.919760 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.919720 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d68d8a-8f84-4419-8344-cfe9d883178e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7766142-lg8t5\" (UID: \"20d68d8a-8f84-4419-8344-cfe9d883178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5" Oct 30 09:56:52.919815 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.919801 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p87nf\" (UniqueName: \"kubernetes.io/projected/20d68d8a-8f84-4419-8344-cfe9d883178e-kube-api-access-p87nf\") pod \"maas-api-revoke-sub-e7766142-lg8t5\" (UID: \"20d68d8a-8f84-4419-8344-cfe9d883178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5" Oct 30 09:56:52.919855 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.919838 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22161d0f-3e3c-44db-acfb-81ac4f26a55c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a710ab-hqk2q\" (UID: \"22161d0f-3e3c-44db-acfb-81ac4f26a55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q" Oct 30 09:56:52.920046 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:52.920024 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2s6p\" (UniqueName: \"kubernetes.io/projected/22161d0f-3e3c-44db-acfb-81ac4f26a55c-kube-api-access-x2s6p\") pod \"maas-api-revoke-sub-61a710ab-hqk2q\" (UID: \"22161d0f-3e3c-44db-acfb-81ac4f26a55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q" Oct 30 09:56:53.021238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.021174 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d68d8a-8f84-4419-8344-cfe9d883178e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7766142-lg8t5\" (UID: \"20d68d8a-8f84-4419-8344-cfe9d883178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5" Oct 30 09:56:53.021238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.021219 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p87nf\" (UniqueName: \"kubernetes.io/projected/20d68d8a-8f84-4419-8344-cfe9d883178e-kube-api-access-p87nf\") pod \"maas-api-revoke-sub-e7766142-lg8t5\" (UID: \"20d68d8a-8f84-4419-8344-cfe9d883178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5" Oct 30 09:56:53.021391 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.021344 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22161d0f-3e3c-44db-acfb-81ac4f26a55c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a710ab-hqk2q\" (UID: \"22161d0f-3e3c-44db-acfb-81ac4f26a55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q" Oct 30 09:56:53.021537 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.021520 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2s6p\" (UniqueName: \"kubernetes.io/projected/22161d0f-3e3c-44db-acfb-81ac4f26a55c-kube-api-access-x2s6p\") pod \"maas-api-revoke-sub-61a710ab-hqk2q\" (UID: \"22161d0f-3e3c-44db-acfb-81ac4f26a55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q" Oct 30 09:56:53.021689 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.021665 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d68d8a-8f84-4419-8344-cfe9d883178e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7766142-lg8t5\" (UID: \"20d68d8a-8f84-4419-8344-cfe9d883178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5" Oct 30 09:56:53.021808 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.021699 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22161d0f-3e3c-44db-acfb-81ac4f26a55c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a710ab-hqk2q\" (UID: \"22161d0f-3e3c-44db-acfb-81ac4f26a55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q" Oct 30 09:56:53.022164 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.022147 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22161d0f-3e3c-44db-acfb-81ac4f26a55c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a710ab-hqk2q\" (UID: \"22161d0f-3e3c-44db-acfb-81ac4f26a55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q" Oct 30 09:56:53.022632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.022610 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d68d8a-8f84-4419-8344-cfe9d883178e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7766142-lg8t5\" (UID: \"20d68d8a-8f84-4419-8344-cfe9d883178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5" Oct 30 09:56:53.024002 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.023972 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d68d8a-8f84-4419-8344-cfe9d883178e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7766142-lg8t5\" (UID: \"20d68d8a-8f84-4419-8344-cfe9d883178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5" Oct 30 09:56:53.024089 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.024071 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22161d0f-3e3c-44db-acfb-81ac4f26a55c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a710ab-hqk2q\" (UID: \"22161d0f-3e3c-44db-acfb-81ac4f26a55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q" Oct 30 09:56:53.041282 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.041261 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p87nf\" (UniqueName: \"kubernetes.io/projected/20d68d8a-8f84-4419-8344-cfe9d883178e-kube-api-access-p87nf\") pod \"maas-api-revoke-sub-e7766142-lg8t5\" (UID: \"20d68d8a-8f84-4419-8344-cfe9d883178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5" Oct 30 09:56:53.041359 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.041309 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2s6p\" (UniqueName: \"kubernetes.io/projected/22161d0f-3e3c-44db-acfb-81ac4f26a55c-kube-api-access-x2s6p\") pod \"maas-api-revoke-sub-61a710ab-hqk2q\" (UID: \"22161d0f-3e3c-44db-acfb-81ac4f26a55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q" Oct 30 09:56:53.160485 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.160463 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5" Oct 30 09:56:53.182678 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.182655 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q" Oct 30 09:56:53.436463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.436439 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn" Oct 30 09:56:53.436664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.436467 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn" event={"ID":"6892707d-6965-4b58-8e66-283364988a03","Type":"ContainerDied","Data":"6de71e25353a978aae65b72e5dfc323b76365a8c02356a1396e501c5fec43599"} Oct 30 09:56:53.436664 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.436516 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6de71e25353a978aae65b72e5dfc323b76365a8c02356a1396e501c5fec43599" Oct 30 09:56:53.465979 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.465948 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5"] Oct 30 09:56:53.466633 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:56:53.466609 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d68d8a_8f84_4419_8344_cfe9d883178e.slice/crio-85efb88e01c308ec86868f19dce842f7a9b205bbc8fd1514aaf20876f4ca99b1 WatchSource:0}: Error finding container 85efb88e01c308ec86868f19dce842f7a9b205bbc8fd1514aaf20876f4ca99b1: Status 404 returned error can't find the container with id 85efb88e01c308ec86868f19dce842f7a9b205bbc8fd1514aaf20876f4ca99b1 Oct 30 09:56:53.501470 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:53.501444 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q"] Oct 30 09:56:53.502393 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:56:53.502364 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22161d0f_3e3c_44db_acfb_81ac4f26a55c.slice/crio-10f47b69afca02070ff0f5604054d2e360710bf492492287ab7d9f39a37eb6b3 WatchSource:0}: Error finding container 10f47b69afca02070ff0f5604054d2e360710bf492492287ab7d9f39a37eb6b3: Status 404 returned error can't find the container with id 10f47b69afca02070ff0f5604054d2e360710bf492492287ab7d9f39a37eb6b3 Oct 30 09:56:54.378977 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:54.378939 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" podUID="c82833df-fd04-431e-8bc2-693a47dfb44e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:56:54.448799 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:54.448767 5938 generic.go:356] "Generic (PLEG): container finished" podID="20d68d8a-8f84-4419-8344-cfe9d883178e" containerID="8cf2558f5bfb696a88b61f4895e4e8714df7f3d4760035ca4ebe8735a019b6dd" exitCode=0 Oct 30 09:56:54.448929 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:54.448902 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5" event={"ID":"20d68d8a-8f84-4419-8344-cfe9d883178e","Type":"ContainerDied","Data":"8cf2558f5bfb696a88b61f4895e4e8714df7f3d4760035ca4ebe8735a019b6dd"} Oct 30 09:56:54.448976 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:54.448960 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5" event={"ID":"20d68d8a-8f84-4419-8344-cfe9d883178e","Type":"ContainerStarted","Data":"85efb88e01c308ec86868f19dce842f7a9b205bbc8fd1514aaf20876f4ca99b1"} Oct 30 09:56:54.450643 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:54.450625 5938 generic.go:356] "Generic (PLEG): container finished" podID="22161d0f-3e3c-44db-acfb-81ac4f26a55c" containerID="8cd1709777137e8758260c3b83422c20526bf52c63d9ffd9e7279a49eccdd723" exitCode=0 Oct 30 09:56:54.450740 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:54.450702 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q" event={"ID":"22161d0f-3e3c-44db-acfb-81ac4f26a55c","Type":"ContainerDied","Data":"8cd1709777137e8758260c3b83422c20526bf52c63d9ffd9e7279a49eccdd723"} Oct 30 09:56:54.450777 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:54.450749 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q" event={"ID":"22161d0f-3e3c-44db-acfb-81ac4f26a55c","Type":"ContainerStarted","Data":"10f47b69afca02070ff0f5604054d2e360710bf492492287ab7d9f39a37eb6b3"} Oct 30 09:56:55.638646 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.637899 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q" Oct 30 09:56:55.685020 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.684988 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5" Oct 30 09:56:55.754155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.754128 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" Oct 30 09:56:55.763243 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.763220 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22161d0f-3e3c-44db-acfb-81ac4f26a55c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22161d0f-3e3c-44db-acfb-81ac4f26a55c-maas-api-openshift-ca\") pod \"22161d0f-3e3c-44db-acfb-81ac4f26a55c\" (UID: \"22161d0f-3e3c-44db-acfb-81ac4f26a55c\") " Oct 30 09:56:55.763403 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.763351 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22161d0f-3e3c-44db-acfb-81ac4f26a55c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22161d0f-3e3c-44db-acfb-81ac4f26a55c-maas-api-xks-ca\") pod \"22161d0f-3e3c-44db-acfb-81ac4f26a55c\" (UID: \"22161d0f-3e3c-44db-acfb-81ac4f26a55c\") " Oct 30 09:56:55.763403 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.763378 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22161d0f-3e3c-44db-acfb-81ac4f26a55c-kube-api-access-x2s6p\" (UniqueName: \"kubernetes.io/projected/22161d0f-3e3c-44db-acfb-81ac4f26a55c-kube-api-access-x2s6p\") pod \"22161d0f-3e3c-44db-acfb-81ac4f26a55c\" (UID: \"22161d0f-3e3c-44db-acfb-81ac4f26a55c\") " Oct 30 09:56:55.763540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.763433 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20d68d8a-8f84-4419-8344-cfe9d883178e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d68d8a-8f84-4419-8344-cfe9d883178e-maas-api-xks-ca\") pod \"20d68d8a-8f84-4419-8344-cfe9d883178e\" (UID: \"20d68d8a-8f84-4419-8344-cfe9d883178e\") " Oct 30 09:56:55.763540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.763450 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d68d8a-8f84-4419-8344-cfe9d883178e-kube-api-access-p87nf\" (UniqueName: \"kubernetes.io/projected/20d68d8a-8f84-4419-8344-cfe9d883178e-kube-api-access-p87nf\") pod \"20d68d8a-8f84-4419-8344-cfe9d883178e\" (UID: \"20d68d8a-8f84-4419-8344-cfe9d883178e\") " Oct 30 09:56:55.763540 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.763469 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d68d8a-8f84-4419-8344-cfe9d883178e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d68d8a-8f84-4419-8344-cfe9d883178e-maas-api-openshift-ca\") pod \"20d68d8a-8f84-4419-8344-cfe9d883178e\" (UID: \"20d68d8a-8f84-4419-8344-cfe9d883178e\") " Oct 30 09:56:55.763761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.763635 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22161d0f-3e3c-44db-acfb-81ac4f26a55c-maas-api-openshift-ca" pod "22161d0f-3e3c-44db-acfb-81ac4f26a55c" (UID: "22161d0f-3e3c-44db-acfb-81ac4f26a55c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:55.764152 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.764125 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d68d8a-8f84-4419-8344-cfe9d883178e-maas-api-openshift-ca" pod "20d68d8a-8f84-4419-8344-cfe9d883178e" (UID: "20d68d8a-8f84-4419-8344-cfe9d883178e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:55.764465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.764443 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22161d0f-3e3c-44db-acfb-81ac4f26a55c-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:55.764465 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.764463 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d68d8a-8f84-4419-8344-cfe9d883178e-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:55.766359 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.766331 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d68d8a-8f84-4419-8344-cfe9d883178e-maas-api-xks-ca" pod "20d68d8a-8f84-4419-8344-cfe9d883178e" (UID: "20d68d8a-8f84-4419-8344-cfe9d883178e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:55.766483 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.766448 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22161d0f-3e3c-44db-acfb-81ac4f26a55c-maas-api-xks-ca" pod "22161d0f-3e3c-44db-acfb-81ac4f26a55c" (UID: "22161d0f-3e3c-44db-acfb-81ac4f26a55c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:55.766548 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.766497 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d68d8a-8f84-4419-8344-cfe9d883178e-kube-api-access-p87nf" pod "20d68d8a-8f84-4419-8344-cfe9d883178e" (UID: "20d68d8a-8f84-4419-8344-cfe9d883178e"). InnerVolumeSpecName "kube-api-access-p87nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:55.766650 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.765886 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22161d0f-3e3c-44db-acfb-81ac4f26a55c-kube-api-access-x2s6p" pod "22161d0f-3e3c-44db-acfb-81ac4f26a55c" (UID: "22161d0f-3e3c-44db-acfb-81ac4f26a55c"). InnerVolumeSpecName "kube-api-access-x2s6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:55.865601 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.865563 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c82833df-fd04-431e-8bc2-693a47dfb44e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82833df-fd04-431e-8bc2-693a47dfb44e-plugins-config-volume\") pod \"c82833df-fd04-431e-8bc2-693a47dfb44e\" (UID: \"c82833df-fd04-431e-8bc2-693a47dfb44e\") " Oct 30 09:56:55.865777 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.865658 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c82833df-fd04-431e-8bc2-693a47dfb44e-kube-api-access-4b9f2\" (UniqueName: \"kubernetes.io/projected/c82833df-fd04-431e-8bc2-693a47dfb44e-kube-api-access-4b9f2\") pod \"c82833df-fd04-431e-8bc2-693a47dfb44e\" (UID: \"c82833df-fd04-431e-8bc2-693a47dfb44e\") " Oct 30 09:56:55.865924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.865900 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82833df-fd04-431e-8bc2-693a47dfb44e-plugins-config-volume" pod "c82833df-fd04-431e-8bc2-693a47dfb44e" (UID: "c82833df-fd04-431e-8bc2-693a47dfb44e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:55.866171 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.866156 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82833df-fd04-431e-8bc2-693a47dfb44e-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:55.866252 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.866172 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22161d0f-3e3c-44db-acfb-81ac4f26a55c-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:55.866252 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.866182 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2s6p\" (UniqueName: \"kubernetes.io/projected/22161d0f-3e3c-44db-acfb-81ac4f26a55c-kube-api-access-x2s6p\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:55.866252 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.866191 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d68d8a-8f84-4419-8344-cfe9d883178e-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:55.866252 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.866199 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p87nf\" (UniqueName: \"kubernetes.io/projected/20d68d8a-8f84-4419-8344-cfe9d883178e-kube-api-access-p87nf\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:55.867974 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.867950 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82833df-fd04-431e-8bc2-693a47dfb44e-kube-api-access-4b9f2" pod "c82833df-fd04-431e-8bc2-693a47dfb44e" (UID: "c82833df-fd04-431e-8bc2-693a47dfb44e"). InnerVolumeSpecName "kube-api-access-4b9f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:55.876114 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.876087 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d"] Oct 30 09:56:55.876781 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.876761 5938 cpu_manager.go:414] "containerMap: removing container" podUID="22161d0f-3e3c-44db-acfb-81ac4f26a55c" containerName="revoke-keys" Oct 30 09:56:55.876885 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.876783 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22161d0f-3e3c-44db-acfb-81ac4f26a55c" containerName="revoke-keys" Oct 30 09:56:55.876885 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.876816 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c82833df-fd04-431e-8bc2-693a47dfb44e" containerName="payload-processing" Oct 30 09:56:55.876885 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.876825 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c82833df-fd04-431e-8bc2-693a47dfb44e" containerName="payload-processing" Oct 30 09:56:55.876983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.876899 5938 cpu_manager.go:414] "containerMap: removing container" podUID="20d68d8a-8f84-4419-8344-cfe9d883178e" containerName="revoke-keys" Oct 30 09:56:55.876983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.876907 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d68d8a-8f84-4419-8344-cfe9d883178e" containerName="revoke-keys" Oct 30 09:56:55.877059 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.877045 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d68d8a-8f84-4419-8344-cfe9d883178e" containerName="revoke-keys" Oct 30 09:56:55.877093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.877067 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="22161d0f-3e3c-44db-acfb-81ac4f26a55c" containerName="revoke-keys" Oct 30 09:56:55.877093 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.877081 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c82833df-fd04-431e-8bc2-693a47dfb44e" containerName="payload-processing" Oct 30 09:56:55.882043 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.882020 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d" Oct 30 09:56:55.906315 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.906288 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d"] Oct 30 09:56:55.967449 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.967420 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtcnq\" (UniqueName: \"kubernetes.io/projected/8be010e8-4a4f-436a-8b6d-615be5470a19-kube-api-access-vtcnq\") pod \"maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d\" (UID: \"8be010e8-4a4f-436a-8b6d-615be5470a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d" Oct 30 09:56:55.967631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.967549 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be010e8-4a4f-436a-8b6d-615be5470a19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d\" (UID: \"8be010e8-4a4f-436a-8b6d-615be5470a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d" Oct 30 09:56:55.967631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.967614 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be010e8-4a4f-436a-8b6d-615be5470a19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d\" (UID: \"8be010e8-4a4f-436a-8b6d-615be5470a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d" Oct 30 09:56:55.967818 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:55.967802 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b9f2\" (UniqueName: \"kubernetes.io/projected/c82833df-fd04-431e-8bc2-693a47dfb44e-kube-api-access-4b9f2\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:56.069260 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.069194 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtcnq\" (UniqueName: \"kubernetes.io/projected/8be010e8-4a4f-436a-8b6d-615be5470a19-kube-api-access-vtcnq\") pod \"maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d\" (UID: \"8be010e8-4a4f-436a-8b6d-615be5470a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d" Oct 30 09:56:56.069407 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.069279 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be010e8-4a4f-436a-8b6d-615be5470a19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d\" (UID: \"8be010e8-4a4f-436a-8b6d-615be5470a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d" Oct 30 09:56:56.069407 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.069305 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be010e8-4a4f-436a-8b6d-615be5470a19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d\" (UID: \"8be010e8-4a4f-436a-8b6d-615be5470a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d" Oct 30 09:56:56.070094 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.070022 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be010e8-4a4f-436a-8b6d-615be5470a19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d\" (UID: \"8be010e8-4a4f-436a-8b6d-615be5470a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d" Oct 30 09:56:56.071876 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.071852 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be010e8-4a4f-436a-8b6d-615be5470a19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d\" (UID: \"8be010e8-4a4f-436a-8b6d-615be5470a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d" Oct 30 09:56:56.090532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.090509 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtcnq\" (UniqueName: \"kubernetes.io/projected/8be010e8-4a4f-436a-8b6d-615be5470a19-kube-api-access-vtcnq\") pod \"maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d\" (UID: \"8be010e8-4a4f-436a-8b6d-615be5470a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d" Oct 30 09:56:56.206470 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.206446 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d" Oct 30 09:56:56.476731 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.476700 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5" Oct 30 09:56:56.477015 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.476723 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5" event={"ID":"20d68d8a-8f84-4419-8344-cfe9d883178e","Type":"ContainerDied","Data":"85efb88e01c308ec86868f19dce842f7a9b205bbc8fd1514aaf20876f4ca99b1"} Oct 30 09:56:56.477158 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.477073 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85efb88e01c308ec86868f19dce842f7a9b205bbc8fd1514aaf20876f4ca99b1" Oct 30 09:56:56.478493 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.478459 5938 generic.go:356] "Generic (PLEG): container finished" podID="c82833df-fd04-431e-8bc2-693a47dfb44e" containerID="6c4589611a6b80b1657e1d7efa51238d5187c448e4a4872336f6709465fc19d7" exitCode=0 Oct 30 09:56:56.478627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.478499 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" Oct 30 09:56:56.478627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.478509 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" event={"ID":"c82833df-fd04-431e-8bc2-693a47dfb44e","Type":"ContainerDied","Data":"6c4589611a6b80b1657e1d7efa51238d5187c448e4a4872336f6709465fc19d7"} Oct 30 09:56:56.478627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.478567 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5" event={"ID":"c82833df-fd04-431e-8bc2-693a47dfb44e","Type":"ContainerDied","Data":"ba14d7d093486bc281cafd444c231cb690ffe86cf45abf971502d5c4d25655f3"} Oct 30 09:56:56.478814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.478631 5938 scope.go:122] "RemoveContainer" containerID="6c4589611a6b80b1657e1d7efa51238d5187c448e4a4872336f6709465fc19d7" Oct 30 09:56:56.480313 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.480283 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q" Oct 30 09:56:56.480313 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.480293 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q" event={"ID":"22161d0f-3e3c-44db-acfb-81ac4f26a55c","Type":"ContainerDied","Data":"10f47b69afca02070ff0f5604054d2e360710bf492492287ab7d9f39a37eb6b3"} Oct 30 09:56:56.480434 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.480332 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10f47b69afca02070ff0f5604054d2e360710bf492492287ab7d9f39a37eb6b3" Oct 30 09:56:56.506223 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.506204 5938 scope.go:122] "RemoveContainer" containerID="6c4589611a6b80b1657e1d7efa51238d5187c448e4a4872336f6709465fc19d7" Oct 30 09:56:56.506499 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:56:56.506478 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4589611a6b80b1657e1d7efa51238d5187c448e4a4872336f6709465fc19d7\": container with ID starting with 6c4589611a6b80b1657e1d7efa51238d5187c448e4a4872336f6709465fc19d7 not found: ID does not exist" containerID="6c4589611a6b80b1657e1d7efa51238d5187c448e4a4872336f6709465fc19d7" Oct 30 09:56:56.506553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.506507 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4589611a6b80b1657e1d7efa51238d5187c448e4a4872336f6709465fc19d7"} err="failed to get container status \"6c4589611a6b80b1657e1d7efa51238d5187c448e4a4872336f6709465fc19d7\": rpc error: code = NotFound desc = could not find container \"6c4589611a6b80b1657e1d7efa51238d5187c448e4a4872336f6709465fc19d7\": container with ID starting with 6c4589611a6b80b1657e1d7efa51238d5187c448e4a4872336f6709465fc19d7 not found: ID does not exist" Oct 30 09:56:56.513788 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.513764 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d"] Oct 30 09:56:56.514298 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:56:56.514265 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be010e8_4a4f_436a_8b6d_615be5470a19.slice/crio-1b105ad24da2d42a599e5af383d8521b41131c78516596c182e777b1ce3d34bd WatchSource:0}: Error finding container 1b105ad24da2d42a599e5af383d8521b41131c78516596c182e777b1ce3d34bd: Status 404 returned error can't find the container with id 1b105ad24da2d42a599e5af383d8521b41131c78516596c182e777b1ce3d34bd Oct 30 09:56:56.577333 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.577230 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5"] Oct 30 09:56:56.596178 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:56.596141 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0dc148-75b8cc658b-r89q5"] Oct 30 09:56:57.490265 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:57.490231 5938 generic.go:356] "Generic (PLEG): container finished" podID="8be010e8-4a4f-436a-8b6d-615be5470a19" containerID="e9308d5de8aff0bd28bc6538ae0421c0c746d9f7d43e5a7d94e2da0909dd154e" exitCode=0 Oct 30 09:56:57.490265 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:57.490264 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d" event={"ID":"8be010e8-4a4f-436a-8b6d-615be5470a19","Type":"ContainerDied","Data":"e9308d5de8aff0bd28bc6538ae0421c0c746d9f7d43e5a7d94e2da0909dd154e"} Oct 30 09:56:57.490730 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:57.490298 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d" event={"ID":"8be010e8-4a4f-436a-8b6d-615be5470a19","Type":"ContainerStarted","Data":"1b105ad24da2d42a599e5af383d8521b41131c78516596c182e777b1ce3d34bd"} Oct 30 09:56:58.257244 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:58.257210 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c82833df-fd04-431e-8bc2-693a47dfb44e" path="/var/lib/kubelet/pods/c82833df-fd04-431e-8bc2-693a47dfb44e/volumes" Oct 30 09:56:58.653566 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:58.653538 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d" Oct 30 09:56:58.806608 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:58.806557 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be010e8-4a4f-436a-8b6d-615be5470a19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be010e8-4a4f-436a-8b6d-615be5470a19-maas-api-openshift-ca\") pod \"8be010e8-4a4f-436a-8b6d-615be5470a19\" (UID: \"8be010e8-4a4f-436a-8b6d-615be5470a19\") " Oct 30 09:56:58.806776 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:58.806728 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8be010e8-4a4f-436a-8b6d-615be5470a19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be010e8-4a4f-436a-8b6d-615be5470a19-maas-api-xks-ca\") pod \"8be010e8-4a4f-436a-8b6d-615be5470a19\" (UID: \"8be010e8-4a4f-436a-8b6d-615be5470a19\") " Oct 30 09:56:58.806864 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:58.806837 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be010e8-4a4f-436a-8b6d-615be5470a19-kube-api-access-vtcnq\" (UniqueName: \"kubernetes.io/projected/8be010e8-4a4f-436a-8b6d-615be5470a19-kube-api-access-vtcnq\") pod \"8be010e8-4a4f-436a-8b6d-615be5470a19\" (UID: \"8be010e8-4a4f-436a-8b6d-615be5470a19\") " Oct 30 09:56:58.806915 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:58.806893 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be010e8-4a4f-436a-8b6d-615be5470a19-maas-api-openshift-ca" pod "8be010e8-4a4f-436a-8b6d-615be5470a19" (UID: "8be010e8-4a4f-436a-8b6d-615be5470a19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:58.807184 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:58.807162 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be010e8-4a4f-436a-8b6d-615be5470a19-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:58.809121 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:58.809070 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be010e8-4a4f-436a-8b6d-615be5470a19-maas-api-xks-ca" pod "8be010e8-4a4f-436a-8b6d-615be5470a19" (UID: "8be010e8-4a4f-436a-8b6d-615be5470a19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:58.809192 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:58.809178 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be010e8-4a4f-436a-8b6d-615be5470a19-kube-api-access-vtcnq" pod "8be010e8-4a4f-436a-8b6d-615be5470a19" (UID: "8be010e8-4a4f-436a-8b6d-615be5470a19"). InnerVolumeSpecName "kube-api-access-vtcnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:58.908868 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:58.908844 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtcnq\" (UniqueName: \"kubernetes.io/projected/8be010e8-4a4f-436a-8b6d-615be5470a19-kube-api-access-vtcnq\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:58.908868 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:58.908867 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be010e8-4a4f-436a-8b6d-615be5470a19-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:56:59.506251 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:59.506222 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d" Oct 30 09:56:59.506442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:59.506244 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d" event={"ID":"8be010e8-4a4f-436a-8b6d-615be5470a19","Type":"ContainerDied","Data":"1b105ad24da2d42a599e5af383d8521b41131c78516596c182e777b1ce3d34bd"} Oct 30 09:56:59.506442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:56:59.506306 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b105ad24da2d42a599e5af383d8521b41131c78516596c182e777b1ce3d34bd" Oct 30 09:57:00.724443 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:00.724413 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8"] Oct 30 09:57:00.737985 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:00.737956 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7"] Oct 30 09:57:03.593978 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:03.593944 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t"] Oct 30 09:57:03.606549 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:03.606525 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a213d-rch2t"] Oct 30 09:57:04.260209 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:04.260180 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5dc127e-23e5-4fac-94f7-6a33c7998515" path="/var/lib/kubelet/pods/a5dc127e-23e5-4fac-94f7-6a33c7998515/volumes" Oct 30 09:57:05.605360 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:05.605326 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm"] Oct 30 09:57:05.904429 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:05.904393 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g"] Oct 30 09:57:05.905069 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:05.905055 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8be010e8-4a4f-436a-8b6d-615be5470a19" containerName="revoke-keys" Oct 30 09:57:05.905128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:05.905070 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be010e8-4a4f-436a-8b6d-615be5470a19" containerName="revoke-keys" Oct 30 09:57:05.905180 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:05.905170 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be010e8-4a4f-436a-8b6d-615be5470a19" containerName="revoke-keys" Oct 30 09:57:05.910051 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:05.910030 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" Oct 30 09:57:05.926155 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:05.926132 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g"] Oct 30 09:57:05.936001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:05.935968 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:57:05.936001 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:05.935970 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:57:06.101079 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:06.101043 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f5bb48-4d18-415e-9033-064e2128d526-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g\" (UID: \"77f5bb48-4d18-415e-9033-064e2128d526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" Oct 30 09:57:06.101278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:06.101096 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn26s\" (UniqueName: \"kubernetes.io/projected/77f5bb48-4d18-415e-9033-064e2128d526-kube-api-access-zn26s\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g\" (UID: \"77f5bb48-4d18-415e-9033-064e2128d526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" Oct 30 09:57:06.101411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:06.101375 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f5bb48-4d18-415e-9033-064e2128d526-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g\" (UID: \"77f5bb48-4d18-415e-9033-064e2128d526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" Oct 30 09:57:06.203539 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:06.203446 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f5bb48-4d18-415e-9033-064e2128d526-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g\" (UID: \"77f5bb48-4d18-415e-9033-064e2128d526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" Oct 30 09:57:06.203744 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:06.203614 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f5bb48-4d18-415e-9033-064e2128d526-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g\" (UID: \"77f5bb48-4d18-415e-9033-064e2128d526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" Oct 30 09:57:06.203744 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:06.203659 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn26s\" (UniqueName: \"kubernetes.io/projected/77f5bb48-4d18-415e-9033-064e2128d526-kube-api-access-zn26s\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g\" (UID: \"77f5bb48-4d18-415e-9033-064e2128d526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" Oct 30 09:57:06.204425 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:06.204401 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f5bb48-4d18-415e-9033-064e2128d526-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g\" (UID: \"77f5bb48-4d18-415e-9033-064e2128d526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" Oct 30 09:57:06.206223 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:06.206204 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f5bb48-4d18-415e-9033-064e2128d526-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g\" (UID: \"77f5bb48-4d18-415e-9033-064e2128d526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" Oct 30 09:57:06.226342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:06.226318 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn26s\" (UniqueName: \"kubernetes.io/projected/77f5bb48-4d18-415e-9033-064e2128d526-kube-api-access-zn26s\") pod \"maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g\" (UID: \"77f5bb48-4d18-415e-9033-064e2128d526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" Oct 30 09:57:06.524451 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:06.524371 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" Oct 30 09:57:06.834735 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:06.834697 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g"] Oct 30 09:57:07.577105 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:07.577030 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" event={"ID":"77f5bb48-4d18-415e-9033-064e2128d526","Type":"ContainerStarted","Data":"d8015538fbe1b266252d67ef77d3a573e458389a14ed009202812f50b2963182"} Oct 30 09:57:07.577105 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:07.577078 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" event={"ID":"77f5bb48-4d18-415e-9033-064e2128d526","Type":"ContainerStarted","Data":"a3594a529984dcbad7afa57c983253db64933e38e7af0f11f6202ce38a062704"} Oct 30 09:57:07.622388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:07.622335 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" podStartSLOduration=2.6223224910000003 podStartE2EDuration="2.622322491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:57:07.6213644 +0000 UTC m=+3745.484070221" watchObservedRunningTime="2026-10-30 09:57:07.622322491 +0000 UTC m=+3745.485028305" Oct 30 09:57:08.585248 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:08.585215 5938 generic.go:356] "Generic (PLEG): container finished" podID="77f5bb48-4d18-415e-9033-064e2128d526" containerID="d8015538fbe1b266252d67ef77d3a573e458389a14ed009202812f50b2963182" exitCode=0 Oct 30 09:57:08.585719 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:08.585297 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" event={"ID":"77f5bb48-4d18-415e-9033-064e2128d526","Type":"ContainerDied","Data":"d8015538fbe1b266252d67ef77d3a573e458389a14ed009202812f50b2963182"} Oct 30 09:57:09.745122 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:09.745099 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" Oct 30 09:57:09.848310 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:09.848229 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f5bb48-4d18-415e-9033-064e2128d526-kube-api-access-zn26s\" (UniqueName: \"kubernetes.io/projected/77f5bb48-4d18-415e-9033-064e2128d526-kube-api-access-zn26s\") pod \"77f5bb48-4d18-415e-9033-064e2128d526\" (UID: \"77f5bb48-4d18-415e-9033-064e2128d526\") " Oct 30 09:57:09.848478 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:09.848342 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77f5bb48-4d18-415e-9033-064e2128d526-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f5bb48-4d18-415e-9033-064e2128d526-maas-api-xks-ca\") pod \"77f5bb48-4d18-415e-9033-064e2128d526\" (UID: \"77f5bb48-4d18-415e-9033-064e2128d526\") " Oct 30 09:57:09.848478 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:09.848465 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f5bb48-4d18-415e-9033-064e2128d526-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f5bb48-4d18-415e-9033-064e2128d526-maas-api-openshift-ca\") pod \"77f5bb48-4d18-415e-9033-064e2128d526\" (UID: \"77f5bb48-4d18-415e-9033-064e2128d526\") " Oct 30 09:57:09.848886 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:09.848843 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f5bb48-4d18-415e-9033-064e2128d526-maas-api-openshift-ca" pod "77f5bb48-4d18-415e-9033-064e2128d526" (UID: "77f5bb48-4d18-415e-9033-064e2128d526"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:09.849235 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:09.849217 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f5bb48-4d18-415e-9033-064e2128d526-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:09.850641 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:09.850581 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f5bb48-4d18-415e-9033-064e2128d526-maas-api-xks-ca" pod "77f5bb48-4d18-415e-9033-064e2128d526" (UID: "77f5bb48-4d18-415e-9033-064e2128d526"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:09.850850 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:09.850828 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f5bb48-4d18-415e-9033-064e2128d526-kube-api-access-zn26s" pod "77f5bb48-4d18-415e-9033-064e2128d526" (UID: "77f5bb48-4d18-415e-9033-064e2128d526"). InnerVolumeSpecName "kube-api-access-zn26s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:09.950280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:09.950253 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn26s\" (UniqueName: \"kubernetes.io/projected/77f5bb48-4d18-415e-9033-064e2128d526-kube-api-access-zn26s\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:09.950280 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:09.950279 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f5bb48-4d18-415e-9033-064e2128d526-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:10.602835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:10.602752 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" Oct 30 09:57:10.602835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:10.602766 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g" event={"ID":"77f5bb48-4d18-415e-9033-064e2128d526","Type":"ContainerDied","Data":"a3594a529984dcbad7afa57c983253db64933e38e7af0f11f6202ce38a062704"} Oct 30 09:57:10.602835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:10.602828 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3594a529984dcbad7afa57c983253db64933e38e7af0f11f6202ce38a062704" Oct 30 09:57:11.124520 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.124486 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq"] Oct 30 09:57:11.125211 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.125194 5938 cpu_manager.go:414] "containerMap: removing container" podUID="77f5bb48-4d18-415e-9033-064e2128d526" containerName="revoke-keys" Oct 30 09:57:11.125291 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.125212 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f5bb48-4d18-415e-9033-064e2128d526" containerName="revoke-keys" Oct 30 09:57:11.125331 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.125316 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f5bb48-4d18-415e-9033-064e2128d526" containerName="revoke-keys" Oct 30 09:57:11.128790 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.128773 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" Oct 30 09:57:11.148276 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.148249 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq"] Oct 30 09:57:11.158095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.158075 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:57:11.158207 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.158135 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:57:11.164829 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.164802 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzw6l\" (UniqueName: \"kubernetes.io/projected/59de32ff-6897-440f-b8da-81ba8ac58f8c-kube-api-access-zzw6l\") pod \"maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq\" (UID: \"59de32ff-6897-440f-b8da-81ba8ac58f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" Oct 30 09:57:11.164965 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.164848 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59de32ff-6897-440f-b8da-81ba8ac58f8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq\" (UID: \"59de32ff-6897-440f-b8da-81ba8ac58f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" Oct 30 09:57:11.165040 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.164974 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59de32ff-6897-440f-b8da-81ba8ac58f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq\" (UID: \"59de32ff-6897-440f-b8da-81ba8ac58f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" Oct 30 09:57:11.266558 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.266529 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59de32ff-6897-440f-b8da-81ba8ac58f8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq\" (UID: \"59de32ff-6897-440f-b8da-81ba8ac58f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" Oct 30 09:57:11.266713 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.266637 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59de32ff-6897-440f-b8da-81ba8ac58f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq\" (UID: \"59de32ff-6897-440f-b8da-81ba8ac58f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" Oct 30 09:57:11.266804 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.266788 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzw6l\" (UniqueName: \"kubernetes.io/projected/59de32ff-6897-440f-b8da-81ba8ac58f8c-kube-api-access-zzw6l\") pod \"maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq\" (UID: \"59de32ff-6897-440f-b8da-81ba8ac58f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" Oct 30 09:57:11.267236 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.267205 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59de32ff-6897-440f-b8da-81ba8ac58f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq\" (UID: \"59de32ff-6897-440f-b8da-81ba8ac58f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" Oct 30 09:57:11.269080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.269060 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59de32ff-6897-440f-b8da-81ba8ac58f8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq\" (UID: \"59de32ff-6897-440f-b8da-81ba8ac58f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" Oct 30 09:57:11.286985 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.286966 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzw6l\" (UniqueName: \"kubernetes.io/projected/59de32ff-6897-440f-b8da-81ba8ac58f8c-kube-api-access-zzw6l\") pod \"maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq\" (UID: \"59de32ff-6897-440f-b8da-81ba8ac58f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" Oct 30 09:57:11.443433 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.443359 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" Oct 30 09:57:11.748787 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:57:11.748754 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59de32ff_6897_440f_b8da_81ba8ac58f8c.slice/crio-c6434e1b4580044e69ecf8acbd40196f714022fd7141a6a58182dbfdc59ae0c2 WatchSource:0}: Error finding container c6434e1b4580044e69ecf8acbd40196f714022fd7141a6a58182dbfdc59ae0c2: Status 404 returned error can't find the container with id c6434e1b4580044e69ecf8acbd40196f714022fd7141a6a58182dbfdc59ae0c2 Oct 30 09:57:11.749621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:11.749578 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq"] Oct 30 09:57:12.633772 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:12.633739 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" event={"ID":"59de32ff-6897-440f-b8da-81ba8ac58f8c","Type":"ContainerStarted","Data":"9984a965f181f7b6c3d2c407cd73c8a559e60e39d41e8e4e6f73fc3a9e5a37ed"} Oct 30 09:57:12.634303 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:12.633797 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" event={"ID":"59de32ff-6897-440f-b8da-81ba8ac58f8c","Type":"ContainerStarted","Data":"c6434e1b4580044e69ecf8acbd40196f714022fd7141a6a58182dbfdc59ae0c2"} Oct 30 09:57:12.677705 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:12.677656 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" podStartSLOduration=1.677641999 podStartE2EDuration="1.677641999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:57:12.673486375 +0000 UTC m=+3750.536192198" watchObservedRunningTime="2026-10-30 09:57:12.677641999 +0000 UTC m=+3750.540347837" Oct 30 09:57:13.642859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:13.642827 5938 generic.go:356] "Generic (PLEG): container finished" podID="59de32ff-6897-440f-b8da-81ba8ac58f8c" containerID="9984a965f181f7b6c3d2c407cd73c8a559e60e39d41e8e4e6f73fc3a9e5a37ed" exitCode=0 Oct 30 09:57:13.643271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:13.642908 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" event={"ID":"59de32ff-6897-440f-b8da-81ba8ac58f8c","Type":"ContainerDied","Data":"9984a965f181f7b6c3d2c407cd73c8a559e60e39d41e8e4e6f73fc3a9e5a37ed"} Oct 30 09:57:14.794914 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:14.794884 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" Oct 30 09:57:14.912955 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:14.912884 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59de32ff-6897-440f-b8da-81ba8ac58f8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59de32ff-6897-440f-b8da-81ba8ac58f8c-maas-api-openshift-ca\") pod \"59de32ff-6897-440f-b8da-81ba8ac58f8c\" (UID: \"59de32ff-6897-440f-b8da-81ba8ac58f8c\") " Oct 30 09:57:14.913128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:14.913014 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59de32ff-6897-440f-b8da-81ba8ac58f8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59de32ff-6897-440f-b8da-81ba8ac58f8c-maas-api-xks-ca\") pod \"59de32ff-6897-440f-b8da-81ba8ac58f8c\" (UID: \"59de32ff-6897-440f-b8da-81ba8ac58f8c\") " Oct 30 09:57:14.913128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:14.913044 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59de32ff-6897-440f-b8da-81ba8ac58f8c-kube-api-access-zzw6l\" (UniqueName: \"kubernetes.io/projected/59de32ff-6897-440f-b8da-81ba8ac58f8c-kube-api-access-zzw6l\") pod \"59de32ff-6897-440f-b8da-81ba8ac58f8c\" (UID: \"59de32ff-6897-440f-b8da-81ba8ac58f8c\") " Oct 30 09:57:14.913283 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:14.913257 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59de32ff-6897-440f-b8da-81ba8ac58f8c-maas-api-openshift-ca" pod "59de32ff-6897-440f-b8da-81ba8ac58f8c" (UID: "59de32ff-6897-440f-b8da-81ba8ac58f8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:14.913512 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:14.913493 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59de32ff-6897-440f-b8da-81ba8ac58f8c-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:14.915164 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:14.915133 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59de32ff-6897-440f-b8da-81ba8ac58f8c-maas-api-xks-ca" pod "59de32ff-6897-440f-b8da-81ba8ac58f8c" (UID: "59de32ff-6897-440f-b8da-81ba8ac58f8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:14.915270 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:14.915227 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59de32ff-6897-440f-b8da-81ba8ac58f8c-kube-api-access-zzw6l" pod "59de32ff-6897-440f-b8da-81ba8ac58f8c" (UID: "59de32ff-6897-440f-b8da-81ba8ac58f8c"). InnerVolumeSpecName "kube-api-access-zzw6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:15.014227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.014199 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59de32ff-6897-440f-b8da-81ba8ac58f8c-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:15.014227 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.014225 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzw6l\" (UniqueName: \"kubernetes.io/projected/59de32ff-6897-440f-b8da-81ba8ac58f8c-kube-api-access-zzw6l\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:15.658089 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.658056 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v"] Oct 30 09:57:15.658772 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.658753 5938 cpu_manager.go:414] "containerMap: removing container" podUID="59de32ff-6897-440f-b8da-81ba8ac58f8c" containerName="revoke-keys" Oct 30 09:57:15.658772 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.658772 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59de32ff-6897-440f-b8da-81ba8ac58f8c" containerName="revoke-keys" Oct 30 09:57:15.658928 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.658917 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="59de32ff-6897-440f-b8da-81ba8ac58f8c" containerName="revoke-keys" Oct 30 09:57:15.662782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.662761 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v" Oct 30 09:57:15.663206 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.663190 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" Oct 30 09:57:15.663301 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.663260 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq" event={"ID":"59de32ff-6897-440f-b8da-81ba8ac58f8c","Type":"ContainerDied","Data":"c6434e1b4580044e69ecf8acbd40196f714022fd7141a6a58182dbfdc59ae0c2"} Oct 30 09:57:15.663359 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.663302 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6434e1b4580044e69ecf8acbd40196f714022fd7141a6a58182dbfdc59ae0c2" Oct 30 09:57:15.677727 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.677707 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb"] Oct 30 09:57:15.683053 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.683032 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb" Oct 30 09:57:15.683744 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.683719 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v"] Oct 30 09:57:15.708192 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.708160 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb"] Oct 30 09:57:15.723671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.723642 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85510896-49d4-4b91-b576-444dd1dba1fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e9fb720-n8cbb\" (UID: \"85510896-49d4-4b91-b576-444dd1dba1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb" Oct 30 09:57:15.723829 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.723783 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85510896-49d4-4b91-b576-444dd1dba1fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e9fb720-n8cbb\" (UID: \"85510896-49d4-4b91-b576-444dd1dba1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb" Oct 30 09:57:15.723904 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.723829 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crq5c\" (UniqueName: \"kubernetes.io/projected/85510896-49d4-4b91-b576-444dd1dba1fe-kube-api-access-crq5c\") pod \"maas-api-revoke-sub-9e9fb720-n8cbb\" (UID: \"85510896-49d4-4b91-b576-444dd1dba1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb" Oct 30 09:57:15.723904 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.723883 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb40e476-zhn6v\" (UID: \"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v" Oct 30 09:57:15.724097 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.724075 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-848qb\" (UniqueName: \"kubernetes.io/projected/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-kube-api-access-848qb\") pod \"maas-api-revoke-sub-eb40e476-zhn6v\" (UID: \"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v" Oct 30 09:57:15.724166 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.724116 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb40e476-zhn6v\" (UID: \"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v" Oct 30 09:57:15.747610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.747564 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" containerID="cri-o://2ad503550e79033bd6d23aef40fbb4df33dd9d6b26d7d74bedcc9589f06d351f" gracePeriod=45 Oct 30 09:57:15.763109 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.763082 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" containerID="cri-o://10f6010d8d7837e4792ac4bcd500fbe263d6fc2c478cee785225a7ad5296278b" gracePeriod=45 Oct 30 09:57:15.817694 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:57:15.817660 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:57:15.825431 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.825404 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-848qb\" (UniqueName: \"kubernetes.io/projected/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-kube-api-access-848qb\") pod \"maas-api-revoke-sub-eb40e476-zhn6v\" (UID: \"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v" Oct 30 09:57:15.825560 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.825437 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb40e476-zhn6v\" (UID: \"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v" Oct 30 09:57:15.825560 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.825541 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85510896-49d4-4b91-b576-444dd1dba1fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e9fb720-n8cbb\" (UID: \"85510896-49d4-4b91-b576-444dd1dba1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb" Oct 30 09:57:15.825749 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.825633 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85510896-49d4-4b91-b576-444dd1dba1fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e9fb720-n8cbb\" (UID: \"85510896-49d4-4b91-b576-444dd1dba1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb" Oct 30 09:57:15.825749 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.825660 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crq5c\" (UniqueName: \"kubernetes.io/projected/85510896-49d4-4b91-b576-444dd1dba1fe-kube-api-access-crq5c\") pod \"maas-api-revoke-sub-9e9fb720-n8cbb\" (UID: \"85510896-49d4-4b91-b576-444dd1dba1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb" Oct 30 09:57:15.825749 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.825702 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb40e476-zhn6v\" (UID: \"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v" Oct 30 09:57:15.826316 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.826292 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb40e476-zhn6v\" (UID: \"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v" Oct 30 09:57:15.826405 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.826348 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85510896-49d4-4b91-b576-444dd1dba1fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e9fb720-n8cbb\" (UID: \"85510896-49d4-4b91-b576-444dd1dba1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb" Oct 30 09:57:15.828083 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.828057 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85510896-49d4-4b91-b576-444dd1dba1fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e9fb720-n8cbb\" (UID: \"85510896-49d4-4b91-b576-444dd1dba1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb" Oct 30 09:57:15.828206 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.828186 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb40e476-zhn6v\" (UID: \"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v" Oct 30 09:57:15.846614 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.846557 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-848qb\" (UniqueName: \"kubernetes.io/projected/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-kube-api-access-848qb\") pod \"maas-api-revoke-sub-eb40e476-zhn6v\" (UID: \"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v" Oct 30 09:57:15.847100 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.847083 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crq5c\" (UniqueName: \"kubernetes.io/projected/85510896-49d4-4b91-b576-444dd1dba1fe-kube-api-access-crq5c\") pod \"maas-api-revoke-sub-9e9fb720-n8cbb\" (UID: \"85510896-49d4-4b91-b576-444dd1dba1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb" Oct 30 09:57:15.873906 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.873884 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr"] Oct 30 09:57:15.896045 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.894542 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz"] Oct 30 09:57:15.988774 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.988745 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v" Oct 30 09:57:15.999062 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:15.999037 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb" Oct 30 09:57:16.138899 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.138498 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:57:16.146641 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.143356 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:57:16.233289 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.233204 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac81131-bcdc-452b-a551-dea9888c9155-kube-api-access-lbx4j\" (UniqueName: \"kubernetes.io/projected/4ac81131-bcdc-452b-a551-dea9888c9155-kube-api-access-lbx4j\") pod \"4ac81131-bcdc-452b-a551-dea9888c9155\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " Oct 30 09:57:16.233464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.233360 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-dshm\") pod \"71e1eff7-c792-4bee-a800-68580ce486f3\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " Oct 30 09:57:16.233464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.233418 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-tmp-dir\") pod \"4ac81131-bcdc-452b-a551-dea9888c9155\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " Oct 30 09:57:16.233464 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.233452 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-model-cache\") pod \"71e1eff7-c792-4bee-a800-68580ce486f3\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " Oct 30 09:57:16.233676 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.233520 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ac81131-bcdc-452b-a551-dea9888c9155-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac81131-bcdc-452b-a551-dea9888c9155-tls-certs\") pod \"4ac81131-bcdc-452b-a551-dea9888c9155\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " Oct 30 09:57:16.233676 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.233555 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-model-cache\") pod \"4ac81131-bcdc-452b-a551-dea9888c9155\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " Oct 30 09:57:16.233802 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.233716 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-tmp-dir" pod "4ac81131-bcdc-452b-a551-dea9888c9155" (UID: "4ac81131-bcdc-452b-a551-dea9888c9155"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:16.233802 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.233725 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e1eff7-c792-4bee-a800-68580ce486f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/71e1eff7-c792-4bee-a800-68580ce486f3-tls-certs\") pod \"71e1eff7-c792-4bee-a800-68580ce486f3\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " Oct 30 09:57:16.233914 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.233830 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-model-cache" pod "71e1eff7-c792-4bee-a800-68580ce486f3" (UID: "71e1eff7-c792-4bee-a800-68580ce486f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:16.233914 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.233851 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-home\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-home\") pod \"71e1eff7-c792-4bee-a800-68580ce486f3\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " Oct 30 09:57:16.234197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.233959 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-dshm\") pod \"4ac81131-bcdc-452b-a551-dea9888c9155\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " Oct 30 09:57:16.234197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.233990 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-home\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-home\") pod \"4ac81131-bcdc-452b-a551-dea9888c9155\" (UID: \"4ac81131-bcdc-452b-a551-dea9888c9155\") " Oct 30 09:57:16.234197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.234012 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-model-cache" pod "4ac81131-bcdc-452b-a551-dea9888c9155" (UID: "4ac81131-bcdc-452b-a551-dea9888c9155"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:16.234197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.234030 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-tmp-dir\") pod \"71e1eff7-c792-4bee-a800-68580ce486f3\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " Oct 30 09:57:16.234197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.234070 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e1eff7-c792-4bee-a800-68580ce486f3-kube-api-access-xtnxg\" (UniqueName: \"kubernetes.io/projected/71e1eff7-c792-4bee-a800-68580ce486f3-kube-api-access-xtnxg\") pod \"71e1eff7-c792-4bee-a800-68580ce486f3\" (UID: \"71e1eff7-c792-4bee-a800-68580ce486f3\") " Oct 30 09:57:16.234542 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.234377 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-home" pod "4ac81131-bcdc-452b-a551-dea9888c9155" (UID: "4ac81131-bcdc-452b-a551-dea9888c9155"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:16.239671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.234630 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-tmp-dir" pod "71e1eff7-c792-4bee-a800-68580ce486f3" (UID: "71e1eff7-c792-4bee-a800-68580ce486f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:16.239671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.234866 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-home" pod "71e1eff7-c792-4bee-a800-68580ce486f3" (UID: "71e1eff7-c792-4bee-a800-68580ce486f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:16.239671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.234959 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.239671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.234981 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.239671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.234993 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.239671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.235003 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.239671 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.235014 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.246349 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.246307 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e1eff7-c792-4bee-a800-68580ce486f3-tls-certs" pod "71e1eff7-c792-4bee-a800-68580ce486f3" (UID: "71e1eff7-c792-4bee-a800-68580ce486f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:16.246349 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.246321 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ac81131-bcdc-452b-a551-dea9888c9155-tls-certs" pod "4ac81131-bcdc-452b-a551-dea9888c9155" (UID: "4ac81131-bcdc-452b-a551-dea9888c9155"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:16.246683 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.246657 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-dshm" pod "71e1eff7-c792-4bee-a800-68580ce486f3" (UID: "71e1eff7-c792-4bee-a800-68580ce486f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:16.246784 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.246684 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e1eff7-c792-4bee-a800-68580ce486f3-kube-api-access-xtnxg" pod "71e1eff7-c792-4bee-a800-68580ce486f3" (UID: "71e1eff7-c792-4bee-a800-68580ce486f3"). InnerVolumeSpecName "kube-api-access-xtnxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:16.247031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.246958 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-dshm" pod "4ac81131-bcdc-452b-a551-dea9888c9155" (UID: "4ac81131-bcdc-452b-a551-dea9888c9155"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:16.247031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.247004 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac81131-bcdc-452b-a551-dea9888c9155-kube-api-access-lbx4j" pod "4ac81131-bcdc-452b-a551-dea9888c9155" (UID: "4ac81131-bcdc-452b-a551-dea9888c9155"). InnerVolumeSpecName "kube-api-access-lbx4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:16.325061 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.325033 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb"] Oct 30 09:57:16.325412 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:57:16.325391 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85510896_49d4_4b91_b576_444dd1dba1fe.slice/crio-43ea287ff7ed0f432566bd48c5154a556f877e5c4e0b5e6464628b7d8ae12f89 WatchSource:0}: Error finding container 43ea287ff7ed0f432566bd48c5154a556f877e5c4e0b5e6464628b7d8ae12f89: Status 404 returned error can't find the container with id 43ea287ff7ed0f432566bd48c5154a556f877e5c4e0b5e6464628b7d8ae12f89 Oct 30 09:57:16.336494 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.336468 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ac81131-bcdc-452b-a551-dea9888c9155-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.336632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.336498 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtnxg\" (UniqueName: \"kubernetes.io/projected/71e1eff7-c792-4bee-a800-68580ce486f3-kube-api-access-xtnxg\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.336632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.336514 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbx4j\" (UniqueName: \"kubernetes.io/projected/4ac81131-bcdc-452b-a551-dea9888c9155-kube-api-access-lbx4j\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.336632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.336527 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.336632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.336548 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac81131-bcdc-452b-a551-dea9888c9155-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.336632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.336561 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71e1eff7-c792-4bee-a800-68580ce486f3-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.336632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.336572 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71e1eff7-c792-4bee-a800-68580ce486f3-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.357063 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.357039 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v"] Oct 30 09:57:16.365857 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:57:16.365831 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9dc48b_8fdb_4ec1_ac2e_c447a2a96a50.slice/crio-29623f0fd243fa9564988123bd71994d38290e3811c22bcabf99b7edb986d613 WatchSource:0}: Error finding container 29623f0fd243fa9564988123bd71994d38290e3811c22bcabf99b7edb986d613: Status 404 returned error can't find the container with id 29623f0fd243fa9564988123bd71994d38290e3811c22bcabf99b7edb986d613 Oct 30 09:57:16.673422 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.673376 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v" event={"ID":"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50","Type":"ContainerStarted","Data":"29623f0fd243fa9564988123bd71994d38290e3811c22bcabf99b7edb986d613"} Oct 30 09:57:16.674837 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.674798 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb" event={"ID":"85510896-49d4-4b91-b576-444dd1dba1fe","Type":"ContainerStarted","Data":"43ea287ff7ed0f432566bd48c5154a556f877e5c4e0b5e6464628b7d8ae12f89"} Oct 30 09:57:16.676412 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.676383 5938 generic.go:356] "Generic (PLEG): container finished" podID="4ac81131-bcdc-452b-a551-dea9888c9155" containerID="10f6010d8d7837e4792ac4bcd500fbe263d6fc2c478cee785225a7ad5296278b" exitCode=0 Oct 30 09:57:16.676572 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.676447 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" Oct 30 09:57:16.676572 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.676474 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" event={"ID":"4ac81131-bcdc-452b-a551-dea9888c9155","Type":"ContainerDied","Data":"10f6010d8d7837e4792ac4bcd500fbe263d6fc2c478cee785225a7ad5296278b"} Oct 30 09:57:16.676572 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.676514 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7" event={"ID":"4ac81131-bcdc-452b-a551-dea9888c9155","Type":"ContainerDied","Data":"29bfe7e0d3989cef3e79f0a459db473ec70a178ce071663831f70418449cfa52"} Oct 30 09:57:16.676572 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.676537 5938 scope.go:122] "RemoveContainer" containerID="10f6010d8d7837e4792ac4bcd500fbe263d6fc2c478cee785225a7ad5296278b" Oct 30 09:57:16.678301 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.678278 5938 generic.go:356] "Generic (PLEG): container finished" podID="71e1eff7-c792-4bee-a800-68580ce486f3" containerID="2ad503550e79033bd6d23aef40fbb4df33dd9d6b26d7d74bedcc9589f06d351f" exitCode=0 Oct 30 09:57:16.678418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.678342 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" Oct 30 09:57:16.678418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.678362 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" event={"ID":"71e1eff7-c792-4bee-a800-68580ce486f3","Type":"ContainerDied","Data":"2ad503550e79033bd6d23aef40fbb4df33dd9d6b26d7d74bedcc9589f06d351f"} Oct 30 09:57:16.678418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.678400 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8" event={"ID":"71e1eff7-c792-4bee-a800-68580ce486f3","Type":"ContainerDied","Data":"5220ee0b504b044c6eda6ab20cfc243968f70a59fb5feeabac758b90a981eb9f"} Oct 30 09:57:16.688351 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.688335 5938 scope.go:122] "RemoveContainer" containerID="10f6010d8d7837e4792ac4bcd500fbe263d6fc2c478cee785225a7ad5296278b" Oct 30 09:57:16.688694 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:57:16.688670 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10f6010d8d7837e4792ac4bcd500fbe263d6fc2c478cee785225a7ad5296278b\": container with ID starting with 10f6010d8d7837e4792ac4bcd500fbe263d6fc2c478cee785225a7ad5296278b not found: ID does not exist" containerID="10f6010d8d7837e4792ac4bcd500fbe263d6fc2c478cee785225a7ad5296278b" Oct 30 09:57:16.688813 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.688701 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10f6010d8d7837e4792ac4bcd500fbe263d6fc2c478cee785225a7ad5296278b"} err="failed to get container status \"10f6010d8d7837e4792ac4bcd500fbe263d6fc2c478cee785225a7ad5296278b\": rpc error: code = NotFound desc = could not find container \"10f6010d8d7837e4792ac4bcd500fbe263d6fc2c478cee785225a7ad5296278b\": container with ID starting with 10f6010d8d7837e4792ac4bcd500fbe263d6fc2c478cee785225a7ad5296278b not found: ID does not exist" Oct 30 09:57:16.688813 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.688719 5938 scope.go:122] "RemoveContainer" containerID="2ad503550e79033bd6d23aef40fbb4df33dd9d6b26d7d74bedcc9589f06d351f" Oct 30 09:57:16.701620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.700684 5938 scope.go:122] "RemoveContainer" containerID="2ad503550e79033bd6d23aef40fbb4df33dd9d6b26d7d74bedcc9589f06d351f" Oct 30 09:57:16.701620 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:57:16.701050 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad503550e79033bd6d23aef40fbb4df33dd9d6b26d7d74bedcc9589f06d351f\": container with ID starting with 2ad503550e79033bd6d23aef40fbb4df33dd9d6b26d7d74bedcc9589f06d351f not found: ID does not exist" containerID="2ad503550e79033bd6d23aef40fbb4df33dd9d6b26d7d74bedcc9589f06d351f" Oct 30 09:57:16.701620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.701111 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad503550e79033bd6d23aef40fbb4df33dd9d6b26d7d74bedcc9589f06d351f"} err="failed to get container status \"2ad503550e79033bd6d23aef40fbb4df33dd9d6b26d7d74bedcc9589f06d351f\": rpc error: code = NotFound desc = could not find container \"2ad503550e79033bd6d23aef40fbb4df33dd9d6b26d7d74bedcc9589f06d351f\": container with ID starting with 2ad503550e79033bd6d23aef40fbb4df33dd9d6b26d7d74bedcc9589f06d351f not found: ID does not exist" Oct 30 09:57:16.732763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.732577 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7"] Oct 30 09:57:16.738264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.738242 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/premium-w2-e3d2ff-kserve-6f6944dfcd-2x7w7"] Oct 30 09:57:16.773697 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.773667 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8"] Oct 30 09:57:16.779924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:16.779904 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/distinct-w2-e3d2ff-kserve-7bb95f665-287z8"] Oct 30 09:57:17.687381 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:17.687345 5938 generic.go:356] "Generic (PLEG): container finished" podID="9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50" containerID="800613d3a983861e5ad4b9ad1f6259c24626e1abaca37e0041c076eade7be584" exitCode=0 Oct 30 09:57:17.687852 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:17.687433 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v" event={"ID":"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50","Type":"ContainerDied","Data":"800613d3a983861e5ad4b9ad1f6259c24626e1abaca37e0041c076eade7be584"} Oct 30 09:57:17.689488 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:17.689461 5938 generic.go:356] "Generic (PLEG): container finished" podID="85510896-49d4-4b91-b576-444dd1dba1fe" containerID="91e0168d6db9878e811f9170eda5aa97691f78d8690f0f873a3794bb06116156" exitCode=0 Oct 30 09:57:17.689632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:17.689499 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb" event={"ID":"85510896-49d4-4b91-b576-444dd1dba1fe","Type":"ContainerDied","Data":"91e0168d6db9878e811f9170eda5aa97691f78d8690f0f873a3794bb06116156"} Oct 30 09:57:18.258411 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.258375 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" path="/var/lib/kubelet/pods/4ac81131-bcdc-452b-a551-dea9888c9155/volumes" Oct 30 09:57:18.258852 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.258837 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" path="/var/lib/kubelet/pods/71e1eff7-c792-4bee-a800-68580ce486f3/volumes" Oct 30 09:57:18.880397 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.880373 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb" Oct 30 09:57:18.883418 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.883393 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v" Oct 30 09:57:18.968412 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.968317 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-maas-api-xks-ca\") pod \"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50\" (UID: \"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50\") " Oct 30 09:57:18.968412 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.968383 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85510896-49d4-4b91-b576-444dd1dba1fe-kube-api-access-crq5c\" (UniqueName: \"kubernetes.io/projected/85510896-49d4-4b91-b576-444dd1dba1fe-kube-api-access-crq5c\") pod \"85510896-49d4-4b91-b576-444dd1dba1fe\" (UID: \"85510896-49d4-4b91-b576-444dd1dba1fe\") " Oct 30 09:57:18.968701 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.968527 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-maas-api-openshift-ca\") pod \"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50\" (UID: \"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50\") " Oct 30 09:57:18.969139 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.969102 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85510896-49d4-4b91-b576-444dd1dba1fe-maas-api-openshift-ca" pod "85510896-49d4-4b91-b576-444dd1dba1fe" (UID: "85510896-49d4-4b91-b576-444dd1dba1fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:18.969297 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.969275 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-maas-api-openshift-ca" pod "9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50" (UID: "9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:18.969381 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.969319 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85510896-49d4-4b91-b576-444dd1dba1fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85510896-49d4-4b91-b576-444dd1dba1fe-maas-api-openshift-ca\") pod \"85510896-49d4-4b91-b576-444dd1dba1fe\" (UID: \"85510896-49d4-4b91-b576-444dd1dba1fe\") " Oct 30 09:57:18.969447 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.969422 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85510896-49d4-4b91-b576-444dd1dba1fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85510896-49d4-4b91-b576-444dd1dba1fe-maas-api-xks-ca\") pod \"85510896-49d4-4b91-b576-444dd1dba1fe\" (UID: \"85510896-49d4-4b91-b576-444dd1dba1fe\") " Oct 30 09:57:18.969967 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.969454 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-kube-api-access-848qb\" (UniqueName: \"kubernetes.io/projected/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-kube-api-access-848qb\") pod \"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50\" (UID: \"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50\") " Oct 30 09:57:18.970359 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.970333 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:18.970456 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.970366 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85510896-49d4-4b91-b576-444dd1dba1fe-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:18.972420 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.972083 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85510896-49d4-4b91-b576-444dd1dba1fe-kube-api-access-crq5c" pod "85510896-49d4-4b91-b576-444dd1dba1fe" (UID: "85510896-49d4-4b91-b576-444dd1dba1fe"). InnerVolumeSpecName "kube-api-access-crq5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:18.973339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.973308 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-maas-api-xks-ca" pod "9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50" (UID: "9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:18.974026 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.973992 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85510896-49d4-4b91-b576-444dd1dba1fe-maas-api-xks-ca" pod "85510896-49d4-4b91-b576-444dd1dba1fe" (UID: "85510896-49d4-4b91-b576-444dd1dba1fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:18.974122 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:18.974098 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-kube-api-access-848qb" pod "9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50" (UID: "9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50"). InnerVolumeSpecName "kube-api-access-848qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:19.071288 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:19.071254 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:19.071288 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:19.071281 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crq5c\" (UniqueName: \"kubernetes.io/projected/85510896-49d4-4b91-b576-444dd1dba1fe-kube-api-access-crq5c\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:19.071288 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:19.071291 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85510896-49d4-4b91-b576-444dd1dba1fe-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:19.071623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:19.071301 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-848qb\" (UniqueName: \"kubernetes.io/projected/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50-kube-api-access-848qb\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:19.552649 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:19.552609 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g"] Oct 30 09:57:19.555768 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:19.555735 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f6ee2-shh2g"] Oct 30 09:57:19.709842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:19.709813 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v" Oct 30 09:57:19.709842 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:19.709840 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v" event={"ID":"9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50","Type":"ContainerDied","Data":"29623f0fd243fa9564988123bd71994d38290e3811c22bcabf99b7edb986d613"} Oct 30 09:57:19.710292 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:19.709897 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29623f0fd243fa9564988123bd71994d38290e3811c22bcabf99b7edb986d613" Oct 30 09:57:19.711706 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:19.711679 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb" event={"ID":"85510896-49d4-4b91-b576-444dd1dba1fe","Type":"ContainerDied","Data":"43ea287ff7ed0f432566bd48c5154a556f877e5c4e0b5e6464628b7d8ae12f89"} Oct 30 09:57:19.711830 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:19.711726 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43ea287ff7ed0f432566bd48c5154a556f877e5c4e0b5e6464628b7d8ae12f89" Oct 30 09:57:19.711830 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:19.711689 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb" Oct 30 09:57:20.069223 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:20.069192 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd"] Oct 30 09:57:20.072328 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:20.072303 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c5447d-cbwnd"] Oct 30 09:57:20.257960 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:20.257929 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f5bb48-4d18-415e-9033-064e2128d526" path="/var/lib/kubelet/pods/77f5bb48-4d18-415e-9033-064e2128d526/volumes" Oct 30 09:57:20.258392 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:20.258378 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c62c859-8cd1-478e-8f0c-d67506b620cd" path="/var/lib/kubelet/pods/9c62c859-8cd1-478e-8f0c-d67506b620cd/volumes" Oct 30 09:57:20.628130 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:20.628092 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" podUID="89b3980d-9c68-495b-a417-ca06457f2ea9" containerName="main" containerID="cri-o://c9fe4791a5cfd7ad14c2e426c1782b8327dda61f375dcb708b7a12d1133522a6" gracePeriod=45 Oct 30 09:57:20.918086 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:20.918056 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:57:21.000338 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.000309 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b3980d-9c68-495b-a417-ca06457f2ea9-kube-api-access-2flms\" (UniqueName: \"kubernetes.io/projected/89b3980d-9c68-495b-a417-ca06457f2ea9-kube-api-access-2flms\") pod \"89b3980d-9c68-495b-a417-ca06457f2ea9\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " Oct 30 09:57:21.000504 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.000363 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-home\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-home\") pod \"89b3980d-9c68-495b-a417-ca06457f2ea9\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " Oct 30 09:57:21.000504 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.000390 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-dshm\") pod \"89b3980d-9c68-495b-a417-ca06457f2ea9\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " Oct 30 09:57:21.000504 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.000448 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b3980d-9c68-495b-a417-ca06457f2ea9-tls-certs\" (UniqueName: \"kubernetes.io/secret/89b3980d-9c68-495b-a417-ca06457f2ea9-tls-certs\") pod \"89b3980d-9c68-495b-a417-ca06457f2ea9\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " Oct 30 09:57:21.000725 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.000553 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-tmp-dir\") pod \"89b3980d-9c68-495b-a417-ca06457f2ea9\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " Oct 30 09:57:21.000725 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.000580 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-model-cache\") pod \"89b3980d-9c68-495b-a417-ca06457f2ea9\" (UID: \"89b3980d-9c68-495b-a417-ca06457f2ea9\") " Oct 30 09:57:21.000725 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.000666 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-home" pod "89b3980d-9c68-495b-a417-ca06457f2ea9" (UID: "89b3980d-9c68-495b-a417-ca06457f2ea9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:21.000917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.000896 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-tmp-dir" pod "89b3980d-9c68-495b-a417-ca06457f2ea9" (UID: "89b3980d-9c68-495b-a417-ca06457f2ea9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:21.001122 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.001094 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-model-cache" pod "89b3980d-9c68-495b-a417-ca06457f2ea9" (UID: "89b3980d-9c68-495b-a417-ca06457f2ea9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:21.001211 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.001159 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:21.001348 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.001259 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:21.002875 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.002854 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-dshm" pod "89b3980d-9c68-495b-a417-ca06457f2ea9" (UID: "89b3980d-9c68-495b-a417-ca06457f2ea9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:21.003131 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.003112 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b3980d-9c68-495b-a417-ca06457f2ea9-tls-certs" pod "89b3980d-9c68-495b-a417-ca06457f2ea9" (UID: "89b3980d-9c68-495b-a417-ca06457f2ea9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:21.003381 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.003367 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b3980d-9c68-495b-a417-ca06457f2ea9-kube-api-access-2flms" pod "89b3980d-9c68-495b-a417-ca06457f2ea9" (UID: "89b3980d-9c68-495b-a417-ca06457f2ea9"). InnerVolumeSpecName "kube-api-access-2flms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:21.102506 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.102485 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2flms\" (UniqueName: \"kubernetes.io/projected/89b3980d-9c68-495b-a417-ca06457f2ea9-kube-api-access-2flms\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:21.102506 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.102507 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:21.102898 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.102516 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b3980d-9c68-495b-a417-ca06457f2ea9-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:21.102898 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.102525 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b3980d-9c68-495b-a417-ca06457f2ea9-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:21.729303 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.729268 5938 generic.go:356] "Generic (PLEG): container finished" podID="89b3980d-9c68-495b-a417-ca06457f2ea9" containerID="c9fe4791a5cfd7ad14c2e426c1782b8327dda61f375dcb708b7a12d1133522a6" exitCode=0 Oct 30 09:57:21.729496 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.729323 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" Oct 30 09:57:21.729496 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.729351 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" event={"ID":"89b3980d-9c68-495b-a417-ca06457f2ea9","Type":"ContainerDied","Data":"c9fe4791a5cfd7ad14c2e426c1782b8327dda61f375dcb708b7a12d1133522a6"} Oct 30 09:57:21.729496 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.729397 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm" event={"ID":"89b3980d-9c68-495b-a417-ca06457f2ea9","Type":"ContainerDied","Data":"a9854053d3f4853839d160f37568402f1cfa582dd2526be6464d089cbc51fb80"} Oct 30 09:57:21.729496 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.729418 5938 scope.go:122] "RemoveContainer" containerID="c9fe4791a5cfd7ad14c2e426c1782b8327dda61f375dcb708b7a12d1133522a6" Oct 30 09:57:21.741181 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.741166 5938 scope.go:122] "RemoveContainer" containerID="c9fe4791a5cfd7ad14c2e426c1782b8327dda61f375dcb708b7a12d1133522a6" Oct 30 09:57:21.741416 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:57:21.741397 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9fe4791a5cfd7ad14c2e426c1782b8327dda61f375dcb708b7a12d1133522a6\": container with ID starting with c9fe4791a5cfd7ad14c2e426c1782b8327dda61f375dcb708b7a12d1133522a6 not found: ID does not exist" containerID="c9fe4791a5cfd7ad14c2e426c1782b8327dda61f375dcb708b7a12d1133522a6" Oct 30 09:57:21.741466 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.741429 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9fe4791a5cfd7ad14c2e426c1782b8327dda61f375dcb708b7a12d1133522a6"} err="failed to get container status \"c9fe4791a5cfd7ad14c2e426c1782b8327dda61f375dcb708b7a12d1133522a6\": rpc error: code = NotFound desc = could not find container \"c9fe4791a5cfd7ad14c2e426c1782b8327dda61f375dcb708b7a12d1133522a6\": container with ID starting with c9fe4791a5cfd7ad14c2e426c1782b8327dda61f375dcb708b7a12d1133522a6 not found: ID does not exist" Oct 30 09:57:21.785194 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.785172 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm"] Oct 30 09:57:21.797657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:21.797634 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-bdd208/test-model-bdd208-kserve-f7ffc85f9-b6lcm"] Oct 30 09:57:22.261599 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:22.261563 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b3980d-9c68-495b-a417-ca06457f2ea9" path="/var/lib/kubelet/pods/89b3980d-9c68-495b-a417-ca06457f2ea9/volumes" Oct 30 09:57:27.227560 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.227521 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524"] Oct 30 09:57:27.228345 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.228321 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" Oct 30 09:57:27.228463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.228346 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" Oct 30 09:57:27.228463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.228397 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50" containerName="revoke-keys" Oct 30 09:57:27.228463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.228405 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50" containerName="revoke-keys" Oct 30 09:57:27.228463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.228417 5938 cpu_manager.go:414] "containerMap: removing container" podUID="89b3980d-9c68-495b-a417-ca06457f2ea9" containerName="main" Oct 30 09:57:27.228463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.228424 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b3980d-9c68-495b-a417-ca06457f2ea9" containerName="main" Oct 30 09:57:27.228463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.228441 5938 cpu_manager.go:414] "containerMap: removing container" podUID="85510896-49d4-4b91-b576-444dd1dba1fe" containerName="revoke-keys" Oct 30 09:57:27.228463 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.228447 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85510896-49d4-4b91-b576-444dd1dba1fe" containerName="revoke-keys" Oct 30 09:57:27.228869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.228470 5938 cpu_manager.go:414] "containerMap: removing container" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" Oct 30 09:57:27.228869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.228478 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" Oct 30 09:57:27.228869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.228641 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac81131-bcdc-452b-a551-dea9888c9155" containerName="main" Oct 30 09:57:27.228869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.228666 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50" containerName="revoke-keys" Oct 30 09:57:27.228869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.228676 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b3980d-9c68-495b-a417-ca06457f2ea9" containerName="main" Oct 30 09:57:27.228869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.228687 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="85510896-49d4-4b91-b576-444dd1dba1fe" containerName="revoke-keys" Oct 30 09:57:27.228869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.228699 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e1eff7-c792-4bee-a800-68580ce486f3" containerName="main" Oct 30 09:57:27.234019 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.233998 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" Oct 30 09:57:27.249921 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.249876 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524"] Oct 30 09:57:27.259408 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.259377 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:57:27.259539 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.259424 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:57:27.393619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.376414 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bdd208-q5524\" (UID: \"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" Oct 30 09:57:27.393619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.376581 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jxbp\" (UniqueName: \"kubernetes.io/projected/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-kube-api-access-7jxbp\") pod \"maas-api-revoke-keys-e2e-inf-b-bdd208-q5524\" (UID: \"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" Oct 30 09:57:27.393619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.376712 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bdd208-q5524\" (UID: \"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" Oct 30 09:57:27.393619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.383467 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz"] Oct 30 09:57:27.393619 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.383710 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" podUID="e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" containerName="istio-proxy" containerID="cri-o://f0b6839f6c78f341a062fba66fc4a213d5a723b7ddf814b28ccf5ed696aabf41" gracePeriod=30 Oct 30 09:57:27.478422 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.478339 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxbp\" (UniqueName: \"kubernetes.io/projected/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-kube-api-access-7jxbp\") pod \"maas-api-revoke-keys-e2e-inf-b-bdd208-q5524\" (UID: \"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" Oct 30 09:57:27.478574 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.478436 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bdd208-q5524\" (UID: \"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" Oct 30 09:57:27.478681 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.478659 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bdd208-q5524\" (UID: \"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" Oct 30 09:57:27.479217 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.479198 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bdd208-q5524\" (UID: \"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" Oct 30 09:57:27.480938 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.480919 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bdd208-q5524\" (UID: \"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" Oct 30 09:57:27.499125 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.499103 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxbp\" (UniqueName: \"kubernetes.io/projected/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-kube-api-access-7jxbp\") pod \"maas-api-revoke-keys-e2e-inf-b-bdd208-q5524\" (UID: \"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" Oct 30 09:57:27.552133 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.551820 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" Oct 30 09:57:27.841620 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.841527 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt"] Oct 30 09:57:27.846978 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.846959 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt" Oct 30 09:57:27.880851 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.880825 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt"] Oct 30 09:57:27.903789 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.903762 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524"] Oct 30 09:57:27.984158 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.984122 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c50a5045-gj7vt\" (UID: \"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt" Oct 30 09:57:27.984158 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.984156 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c50a5045-gj7vt\" (UID: \"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt" Oct 30 09:57:27.984326 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:27.984304 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht44s\" (UniqueName: \"kubernetes.io/projected/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-kube-api-access-ht44s\") pod \"maas-api-revoke-sub-c50a5045-gj7vt\" (UID: \"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt" Oct 30 09:57:28.086033 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:28.085998 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht44s\" (UniqueName: \"kubernetes.io/projected/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-kube-api-access-ht44s\") pod \"maas-api-revoke-sub-c50a5045-gj7vt\" (UID: \"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt" Oct 30 09:57:28.086175 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:28.086158 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c50a5045-gj7vt\" (UID: \"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt" Oct 30 09:57:28.086261 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:28.086184 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c50a5045-gj7vt\" (UID: \"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt" Oct 30 09:57:28.086788 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:28.086767 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c50a5045-gj7vt\" (UID: \"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt" Oct 30 09:57:28.088744 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:28.088724 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c50a5045-gj7vt\" (UID: \"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt" Oct 30 09:57:28.106156 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:28.106109 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht44s\" (UniqueName: \"kubernetes.io/projected/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-kube-api-access-ht44s\") pod \"maas-api-revoke-sub-c50a5045-gj7vt\" (UID: \"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt" Oct 30 09:57:28.161502 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:28.161481 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt" Oct 30 09:57:28.470755 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:28.470730 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt"] Oct 30 09:57:28.471709 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:57:28.471684 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8e41b9_4e7b_4df9_9a2c_80289cbe4cef.slice/crio-545b0a3a8693b349520c98a89bfb802116e0b74fe69b6cdbd35f285bdf6f593e WatchSource:0}: Error finding container 545b0a3a8693b349520c98a89bfb802116e0b74fe69b6cdbd35f285bdf6f593e: Status 404 returned error can't find the container with id 545b0a3a8693b349520c98a89bfb802116e0b74fe69b6cdbd35f285bdf6f593e Oct 30 09:57:28.787415 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:28.787380 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" event={"ID":"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659","Type":"ContainerStarted","Data":"29bde7c1ff6649b9a4d61f297f05c722df4722010e4ad87af6512141df9ededf"} Oct 30 09:57:28.787577 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:28.787441 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" event={"ID":"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659","Type":"ContainerStarted","Data":"9b1e2dad2c95afbbf95ed8ad67133aa48f96ae1b00e65a17b48ee8db63a32bab"} Oct 30 09:57:28.788975 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:28.788952 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt" event={"ID":"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef","Type":"ContainerStarted","Data":"545b0a3a8693b349520c98a89bfb802116e0b74fe69b6cdbd35f285bdf6f593e"} Oct 30 09:57:28.829045 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:28.828995 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" podStartSLOduration=1.828977571 podStartE2EDuration="1.828977571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:57:28.827333199 +0000 UTC m=+3766.690039019" watchObservedRunningTime="2026-10-30 09:57:28.828977571 +0000 UTC m=+3766.691683392" Oct 30 09:57:29.802496 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:29.802415 5938 generic.go:356] "Generic (PLEG): container finished" podID="8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef" containerID="7716044eb7e1047ccb4c2e43bef22d5dc3528b3c65a4d023864157937664303c" exitCode=0 Oct 30 09:57:29.802981 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:29.802525 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt" event={"ID":"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef","Type":"ContainerDied","Data":"7716044eb7e1047ccb4c2e43bef22d5dc3528b3c65a4d023864157937664303c"} Oct 30 09:57:29.804224 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:29.804202 5938 generic.go:356] "Generic (PLEG): container finished" podID="8621f4c3-0a60-4fdb-ab7a-f520bfc3e659" containerID="29bde7c1ff6649b9a4d61f297f05c722df4722010e4ad87af6512141df9ededf" exitCode=0 Oct 30 09:57:29.804336 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:29.804254 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" event={"ID":"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659","Type":"ContainerDied","Data":"29bde7c1ff6649b9a4d61f297f05c722df4722010e4ad87af6512141df9ededf"} Oct 30 09:57:29.977694 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:29.977660 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:57:29.977846 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:29.977721 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" podUID="e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:57:30.905169 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:30.905130 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" containerID="cri-o://c0c5cae79b865713f96c761b3eb1c9f428dadaec9cd095468cb1fe75fd9d9471" gracePeriod=45 Oct 30 09:57:30.922532 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:30.922463 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerName="main" containerID="cri-o://37e85daa004f05bc9952aea7e81d120dcfcb632d1ce53bdae62cf3787b4186bb" gracePeriod=45 Oct 30 09:57:31.068230 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.068162 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" Oct 30 09:57:31.072681 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.072558 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt" Oct 30 09:57:31.088492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.088451 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": dial tcp 10.134.0.39:8000: connect: connection refused" Oct 30 09:57:31.190627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.190601 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:57:31.218415 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.218389 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:57:31.222491 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.222470 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-maas-api-openshift-ca\") pod \"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659\" (UID: \"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659\") " Oct 30 09:57:31.222633 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.222577 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-maas-api-openshift-ca\") pod \"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef\" (UID: \"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef\") " Oct 30 09:57:31.222713 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.222657 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-kube-api-access-ht44s\" (UniqueName: \"kubernetes.io/projected/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-kube-api-access-ht44s\") pod \"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef\" (UID: \"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef\") " Oct 30 09:57:31.222713 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.222699 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-maas-api-xks-ca\") pod \"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef\" (UID: \"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef\") " Oct 30 09:57:31.222823 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.222796 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-kube-api-access-7jxbp\" (UniqueName: \"kubernetes.io/projected/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-kube-api-access-7jxbp\") pod \"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659\" (UID: \"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659\") " Oct 30 09:57:31.222889 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.222874 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-maas-api-xks-ca\") pod \"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659\" (UID: \"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659\") " Oct 30 09:57:31.222953 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.222881 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-maas-api-openshift-ca" pod "8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef" (UID: "8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:31.222953 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.222881 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-maas-api-openshift-ca" pod "8621f4c3-0a60-4fdb-ab7a-f520bfc3e659" (UID: "8621f4c3-0a60-4fdb-ab7a-f520bfc3e659"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:31.223673 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.223649 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.223808 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.223678 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.225031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.225005 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-maas-api-xks-ca" pod "8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef" (UID: "8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:31.225132 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.225060 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-maas-api-xks-ca" pod "8621f4c3-0a60-4fdb-ab7a-f520bfc3e659" (UID: "8621f4c3-0a60-4fdb-ab7a-f520bfc3e659"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:31.225132 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.225081 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-kube-api-access-ht44s" pod "8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef" (UID: "8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef"). InnerVolumeSpecName "kube-api-access-ht44s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:31.225132 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.225109 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-kube-api-access-7jxbp" pod "8621f4c3-0a60-4fdb-ab7a-f520bfc3e659" (UID: "8621f4c3-0a60-4fdb-ab7a-f520bfc3e659"). InnerVolumeSpecName "kube-api-access-7jxbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:31.324794 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.324721 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-home\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-home\") pod \"f900f402-3458-4a8e-aa3b-a92b7ede222f\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " Oct 30 09:57:31.325103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.324821 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-model-cache\") pod \"8ded0c49-3011-4730-af91-f1700a3034b3\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " Oct 30 09:57:31.325103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.324885 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-tmp-dir\") pod \"f900f402-3458-4a8e-aa3b-a92b7ede222f\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " Oct 30 09:57:31.325103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.324921 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-home\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-home\") pod \"8ded0c49-3011-4730-af91-f1700a3034b3\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " Oct 30 09:57:31.325103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.325002 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f900f402-3458-4a8e-aa3b-a92b7ede222f-kube-api-access-8k5z9\" (UniqueName: \"kubernetes.io/projected/f900f402-3458-4a8e-aa3b-a92b7ede222f-kube-api-access-8k5z9\") pod \"f900f402-3458-4a8e-aa3b-a92b7ede222f\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " Oct 30 09:57:31.325103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.325042 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-model-cache\") pod \"f900f402-3458-4a8e-aa3b-a92b7ede222f\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " Oct 30 09:57:31.325103 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.325017 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-home" pod "f900f402-3458-4a8e-aa3b-a92b7ede222f" (UID: "f900f402-3458-4a8e-aa3b-a92b7ede222f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.325382 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.325108 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-dshm\") pod \"8ded0c49-3011-4730-af91-f1700a3034b3\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " Oct 30 09:57:31.325382 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.325133 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-model-cache" pod "8ded0c49-3011-4730-af91-f1700a3034b3" (UID: "8ded0c49-3011-4730-af91-f1700a3034b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.325382 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.325192 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-tmp-dir" pod "f900f402-3458-4a8e-aa3b-a92b7ede222f" (UID: "f900f402-3458-4a8e-aa3b-a92b7ede222f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.325382 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.325343 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-home" pod "8ded0c49-3011-4730-af91-f1700a3034b3" (UID: "8ded0c49-3011-4730-af91-f1700a3034b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.325553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.325406 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-model-cache" pod "f900f402-3458-4a8e-aa3b-a92b7ede222f" (UID: "f900f402-3458-4a8e-aa3b-a92b7ede222f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.325553 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.325472 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-tmp-dir\") pod \"8ded0c49-3011-4730-af91-f1700a3034b3\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " Oct 30 09:57:31.325693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.325560 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ded0c49-3011-4730-af91-f1700a3034b3-kube-api-access-vxn8p\" (UniqueName: \"kubernetes.io/projected/8ded0c49-3011-4730-af91-f1700a3034b3-kube-api-access-vxn8p\") pod \"8ded0c49-3011-4730-af91-f1700a3034b3\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " Oct 30 09:57:31.325693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.325663 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-dshm\") pod \"f900f402-3458-4a8e-aa3b-a92b7ede222f\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " Oct 30 09:57:31.325810 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.325729 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f900f402-3458-4a8e-aa3b-a92b7ede222f-tls-certs\" (UniqueName: \"kubernetes.io/secret/f900f402-3458-4a8e-aa3b-a92b7ede222f-tls-certs\") pod \"f900f402-3458-4a8e-aa3b-a92b7ede222f\" (UID: \"f900f402-3458-4a8e-aa3b-a92b7ede222f\") " Oct 30 09:57:31.325810 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.325781 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ded0c49-3011-4730-af91-f1700a3034b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ded0c49-3011-4730-af91-f1700a3034b3-tls-certs\") pod \"8ded0c49-3011-4730-af91-f1700a3034b3\" (UID: \"8ded0c49-3011-4730-af91-f1700a3034b3\") " Oct 30 09:57:31.325923 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.325842 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-tmp-dir" pod "8ded0c49-3011-4730-af91-f1700a3034b3" (UID: "8ded0c49-3011-4730-af91-f1700a3034b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.327058 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.327028 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.327191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.327072 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.327191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.327088 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.327191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.327104 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht44s\" (UniqueName: \"kubernetes.io/projected/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-kube-api-access-ht44s\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.327191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.327116 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.327191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.327130 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.327191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.327143 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.327191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.327156 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.327191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.327170 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.327191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.327183 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jxbp\" (UniqueName: \"kubernetes.io/projected/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659-kube-api-access-7jxbp\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.328300 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.328272 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f900f402-3458-4a8e-aa3b-a92b7ede222f-kube-api-access-8k5z9" pod "f900f402-3458-4a8e-aa3b-a92b7ede222f" (UID: "f900f402-3458-4a8e-aa3b-a92b7ede222f"). InnerVolumeSpecName "kube-api-access-8k5z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:31.328547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.328525 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ded0c49-3011-4730-af91-f1700a3034b3-kube-api-access-vxn8p" pod "8ded0c49-3011-4730-af91-f1700a3034b3" (UID: "8ded0c49-3011-4730-af91-f1700a3034b3"). InnerVolumeSpecName "kube-api-access-vxn8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:31.328934 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.328911 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-dshm" pod "8ded0c49-3011-4730-af91-f1700a3034b3" (UID: "8ded0c49-3011-4730-af91-f1700a3034b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.329031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.329006 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ded0c49-3011-4730-af91-f1700a3034b3-tls-certs" pod "8ded0c49-3011-4730-af91-f1700a3034b3" (UID: "8ded0c49-3011-4730-af91-f1700a3034b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:31.329290 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.329272 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f900f402-3458-4a8e-aa3b-a92b7ede222f-tls-certs" pod "f900f402-3458-4a8e-aa3b-a92b7ede222f" (UID: "f900f402-3458-4a8e-aa3b-a92b7ede222f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:31.329364 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.329342 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-dshm" pod "f900f402-3458-4a8e-aa3b-a92b7ede222f" (UID: "f900f402-3458-4a8e-aa3b-a92b7ede222f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.428685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.428656 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k5z9\" (UniqueName: \"kubernetes.io/projected/f900f402-3458-4a8e-aa3b-a92b7ede222f-kube-api-access-8k5z9\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.428685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.428682 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ded0c49-3011-4730-af91-f1700a3034b3-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.428685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.428691 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxn8p\" (UniqueName: \"kubernetes.io/projected/8ded0c49-3011-4730-af91-f1700a3034b3-kube-api-access-vxn8p\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.428889 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.428701 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f900f402-3458-4a8e-aa3b-a92b7ede222f-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.428889 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.428710 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f900f402-3458-4a8e-aa3b-a92b7ede222f-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.428889 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.428719 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ded0c49-3011-4730-af91-f1700a3034b3-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.819730 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.819704 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" Oct 30 09:57:31.819928 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.819740 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524" event={"ID":"8621f4c3-0a60-4fdb-ab7a-f520bfc3e659","Type":"ContainerDied","Data":"9b1e2dad2c95afbbf95ed8ad67133aa48f96ae1b00e65a17b48ee8db63a32bab"} Oct 30 09:57:31.819928 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.819798 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b1e2dad2c95afbbf95ed8ad67133aa48f96ae1b00e65a17b48ee8db63a32bab" Oct 30 09:57:31.821262 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.821237 5938 generic.go:356] "Generic (PLEG): container finished" podID="8ded0c49-3011-4730-af91-f1700a3034b3" containerID="c0c5cae79b865713f96c761b3eb1c9f428dadaec9cd095468cb1fe75fd9d9471" exitCode=0 Oct 30 09:57:31.821415 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.821275 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" Oct 30 09:57:31.821415 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.821310 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" event={"ID":"8ded0c49-3011-4730-af91-f1700a3034b3","Type":"ContainerDied","Data":"c0c5cae79b865713f96c761b3eb1c9f428dadaec9cd095468cb1fe75fd9d9471"} Oct 30 09:57:31.821415 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.821347 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr" event={"ID":"8ded0c49-3011-4730-af91-f1700a3034b3","Type":"ContainerDied","Data":"44acef725c84293227c0a78e3575ba2ec6b1c4a8a7fe1012a274bd5957f0da64"} Oct 30 09:57:31.821415 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.821365 5938 scope.go:122] "RemoveContainer" containerID="c0c5cae79b865713f96c761b3eb1c9f428dadaec9cd095468cb1fe75fd9d9471" Oct 30 09:57:31.822939 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.822918 5938 generic.go:356] "Generic (PLEG): container finished" podID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerID="37e85daa004f05bc9952aea7e81d120dcfcb632d1ce53bdae62cf3787b4186bb" exitCode=0 Oct 30 09:57:31.823058 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.822953 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" Oct 30 09:57:31.823124 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.823010 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" event={"ID":"f900f402-3458-4a8e-aa3b-a92b7ede222f","Type":"ContainerDied","Data":"37e85daa004f05bc9952aea7e81d120dcfcb632d1ce53bdae62cf3787b4186bb"} Oct 30 09:57:31.823225 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.823208 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz" event={"ID":"f900f402-3458-4a8e-aa3b-a92b7ede222f","Type":"ContainerDied","Data":"5ca18b98ef3cd46465d72c9808fd9cf065fb9c12cf5c8f86551803c83c9ad402"} Oct 30 09:57:31.826029 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.826002 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt" Oct 30 09:57:31.826172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.826042 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt" event={"ID":"8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef","Type":"ContainerDied","Data":"545b0a3a8693b349520c98a89bfb802116e0b74fe69b6cdbd35f285bdf6f593e"} Oct 30 09:57:31.826172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.826089 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="545b0a3a8693b349520c98a89bfb802116e0b74fe69b6cdbd35f285bdf6f593e" Oct 30 09:57:31.836849 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.836794 5938 scope.go:122] "RemoveContainer" containerID="c0c5cae79b865713f96c761b3eb1c9f428dadaec9cd095468cb1fe75fd9d9471" Oct 30 09:57:31.837126 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:57:31.837101 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c5cae79b865713f96c761b3eb1c9f428dadaec9cd095468cb1fe75fd9d9471\": container with ID starting with c0c5cae79b865713f96c761b3eb1c9f428dadaec9cd095468cb1fe75fd9d9471 not found: ID does not exist" containerID="c0c5cae79b865713f96c761b3eb1c9f428dadaec9cd095468cb1fe75fd9d9471" Oct 30 09:57:31.837214 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.837137 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c5cae79b865713f96c761b3eb1c9f428dadaec9cd095468cb1fe75fd9d9471"} err="failed to get container status \"c0c5cae79b865713f96c761b3eb1c9f428dadaec9cd095468cb1fe75fd9d9471\": rpc error: code = NotFound desc = could not find container \"c0c5cae79b865713f96c761b3eb1c9f428dadaec9cd095468cb1fe75fd9d9471\": container with ID starting with c0c5cae79b865713f96c761b3eb1c9f428dadaec9cd095468cb1fe75fd9d9471 not found: ID does not exist" Oct 30 09:57:31.837214 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.837155 5938 scope.go:122] "RemoveContainer" containerID="37e85daa004f05bc9952aea7e81d120dcfcb632d1ce53bdae62cf3787b4186bb" Oct 30 09:57:31.846732 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.846717 5938 scope.go:122] "RemoveContainer" containerID="37e85daa004f05bc9952aea7e81d120dcfcb632d1ce53bdae62cf3787b4186bb" Oct 30 09:57:31.847011 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:57:31.846995 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e85daa004f05bc9952aea7e81d120dcfcb632d1ce53bdae62cf3787b4186bb\": container with ID starting with 37e85daa004f05bc9952aea7e81d120dcfcb632d1ce53bdae62cf3787b4186bb not found: ID does not exist" containerID="37e85daa004f05bc9952aea7e81d120dcfcb632d1ce53bdae62cf3787b4186bb" Oct 30 09:57:31.847068 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.847016 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e85daa004f05bc9952aea7e81d120dcfcb632d1ce53bdae62cf3787b4186bb"} err="failed to get container status \"37e85daa004f05bc9952aea7e81d120dcfcb632d1ce53bdae62cf3787b4186bb\": rpc error: code = NotFound desc = could not find container \"37e85daa004f05bc9952aea7e81d120dcfcb632d1ce53bdae62cf3787b4186bb\": container with ID starting with 37e85daa004f05bc9952aea7e81d120dcfcb632d1ce53bdae62cf3787b4186bb not found: ID does not exist" Oct 30 09:57:31.877651 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.877625 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr"] Oct 30 09:57:31.882406 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.882386 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/premium-w3-176b6c-kserve-c889c986-jw6fr"] Oct 30 09:57:31.935404 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.935385 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz"] Oct 30 09:57:31.939177 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:31.939158 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/simulator-w3-176b6c-kserve-86db7cfcf-xh6cz"] Oct 30 09:57:32.257424 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.257396 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" path="/var/lib/kubelet/pods/8ded0c49-3011-4730-af91-f1700a3034b3/volumes" Oct 30 09:57:32.257858 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.257844 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" path="/var/lib/kubelet/pods/f900f402-3458-4a8e-aa3b-a92b7ede222f/volumes" Oct 30 09:57:32.646913 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.646891 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:57:32.747612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.743315 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-envoy\") pod \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " Oct 30 09:57:32.747612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.743385 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istiod-ca-cert\") pod \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " Oct 30 09:57:32.747612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.743442 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-token\" (UniqueName: \"kubernetes.io/projected/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-token\") pod \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " Oct 30 09:57:32.747612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.743490 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-workload-socket\") pod \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " Oct 30 09:57:32.747612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.743618 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-credential-socket\") pod \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " Oct 30 09:57:32.747612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.743718 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-data\") pod \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " Oct 30 09:57:32.747612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.743778 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-podinfo\") pod \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " Oct 30 09:57:32.747612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.743865 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-workload-certs\") pod \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " Oct 30 09:57:32.747612 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.743947 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-kube-api-access-7657r\" (UniqueName: \"kubernetes.io/projected/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-kube-api-access-7657r\") pod \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\" (UID: \"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d\") " Oct 30 09:57:32.759546 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.752766 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istiod-ca-cert" pod "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" (UID: "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:32.765611 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.765549 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-workload-socket" pod "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" (UID: "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:32.766038 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.765947 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-token" pod "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" (UID: "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:32.766038 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.765949 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-credential-socket" pod "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" (UID: "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:32.766305 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.766270 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-data" pod "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" (UID: "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:32.766431 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.766348 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-workload-certs" pod "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" (UID: "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:32.766491 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.766444 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-kube-api-access-7657r" pod "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" (UID: "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d"). InnerVolumeSpecName "kube-api-access-7657r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:32.766572 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.766559 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-envoy" pod "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" (UID: "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:32.768872 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.768846 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-podinfo" pod "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" (UID: "e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:57:32.838002 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.837973 5938 generic.go:356] "Generic (PLEG): container finished" podID="e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" containerID="f0b6839f6c78f341a062fba66fc4a213d5a723b7ddf814b28ccf5ed696aabf41" exitCode=0 Oct 30 09:57:32.838143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.838027 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" Oct 30 09:57:32.838143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.838051 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" event={"ID":"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d","Type":"ContainerDied","Data":"f0b6839f6c78f341a062fba66fc4a213d5a723b7ddf814b28ccf5ed696aabf41"} Oct 30 09:57:32.838143 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.838112 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz" event={"ID":"e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d","Type":"ContainerDied","Data":"3bd6aa06018ce7362e2d3a6e96bc616e70f50ca5fb594b94383d316f5bb74d26"} Oct 30 09:57:32.838314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.838143 5938 scope.go:122] "RemoveContainer" containerID="f0b6839f6c78f341a062fba66fc4a213d5a723b7ddf814b28ccf5ed696aabf41" Oct 30 09:57:32.845666 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.845645 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.845666 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.845665 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.845859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.845674 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.845859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.845685 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.845859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.845694 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.845859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.845701 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.845859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.845709 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7657r\" (UniqueName: \"kubernetes.io/projected/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-kube-api-access-7657r\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.845859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.845717 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.845859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.845725 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.849335 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.849319 5938 scope.go:122] "RemoveContainer" containerID="f0b6839f6c78f341a062fba66fc4a213d5a723b7ddf814b28ccf5ed696aabf41" Oct 30 09:57:32.849657 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:57:32.849637 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b6839f6c78f341a062fba66fc4a213d5a723b7ddf814b28ccf5ed696aabf41\": container with ID starting with f0b6839f6c78f341a062fba66fc4a213d5a723b7ddf814b28ccf5ed696aabf41 not found: ID does not exist" containerID="f0b6839f6c78f341a062fba66fc4a213d5a723b7ddf814b28ccf5ed696aabf41" Oct 30 09:57:32.849718 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.849666 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b6839f6c78f341a062fba66fc4a213d5a723b7ddf814b28ccf5ed696aabf41"} err="failed to get container status \"f0b6839f6c78f341a062fba66fc4a213d5a723b7ddf814b28ccf5ed696aabf41\": rpc error: code = NotFound desc = could not find container \"f0b6839f6c78f341a062fba66fc4a213d5a723b7ddf814b28ccf5ed696aabf41\": container with ID starting with f0b6839f6c78f341a062fba66fc4a213d5a723b7ddf814b28ccf5ed696aabf41 not found: ID does not exist" Oct 30 09:57:32.899189 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.899162 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz"] Oct 30 09:57:32.902197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:32.902170 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-zglzz"] Oct 30 09:57:34.257206 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:34.257166 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" path="/var/lib/kubelet/pods/e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d/volumes" Oct 30 09:57:40.189659 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:40.189623 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d"] Oct 30 09:57:40.202887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:40.202847 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e3d2ff-59q9d"] Oct 30 09:57:40.266400 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:40.266362 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be010e8-4a4f-436a-8b6d-615be5470a19" path="/var/lib/kubelet/pods/8be010e8-4a4f-436a-8b6d-615be5470a19/volumes" Oct 30 09:57:44.645499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:44.645468 5938 scope.go:122] "RemoveContainer" containerID="5ea2241db578f7175a417121e5e6bfcc2c2b7b2f98b49430b07cfd7201671bf8" Oct 30 09:57:45.811255 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:57:45.811220 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:57:47.442656 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:47.442620 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm"] Oct 30 09:57:47.443470 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:47.443434 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" podUID="e40f5ce9-777c-44d7-8325-6bb672235db7" containerName="istio-proxy" containerID="cri-o://232d928ec1daf5303c9f5aa0908a2bad62550ae6110b7ae0275d0bfbc071fd99" gracePeriod=30 Oct 30 09:57:49.031314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:49.031274 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524"] Oct 30 09:57:49.044733 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:49.044697 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdd208-q5524"] Oct 30 09:57:50.257740 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:50.257700 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8621f4c3-0a60-4fdb-ab7a-f520bfc3e659" path="/var/lib/kubelet/pods/8621f4c3-0a60-4fdb-ab7a-f520bfc3e659/volumes" Oct 30 09:57:51.072361 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:51.072321 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq"] Oct 30 09:57:51.086314 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:51.086282 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-176b6c-7xlfq"] Oct 30 09:57:52.026454 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.026417 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds"] Oct 30 09:57:52.039396 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.039367 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b4d96e-h4kds"] Oct 30 09:57:52.257847 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.257776 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59de32ff-6897-440f-b8da-81ba8ac58f8c" path="/var/lib/kubelet/pods/59de32ff-6897-440f-b8da-81ba8ac58f8c/volumes" Oct 30 09:57:52.258264 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.258248 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66a46a6f-5252-4a98-85be-982321aab480" path="/var/lib/kubelet/pods/66a46a6f-5252-4a98-85be-982321aab480/volumes" Oct 30 09:57:52.736185 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.735802 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:57:52.770493 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.770451 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e40f5ce9-777c-44d7-8325-6bb672235db7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40f5ce9-777c-44d7-8325-6bb672235db7-istiod-ca-cert\") pod \"e40f5ce9-777c-44d7-8325-6bb672235db7\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " Oct 30 09:57:52.770778 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.770508 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-token\" (UniqueName: \"kubernetes.io/projected/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-token\") pod \"e40f5ce9-777c-44d7-8325-6bb672235db7\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " Oct 30 09:57:52.770778 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.770565 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-workload-socket\") pod \"e40f5ce9-777c-44d7-8325-6bb672235db7\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " Oct 30 09:57:52.770778 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.770667 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40f5ce9-777c-44d7-8325-6bb672235db7-kube-api-access-mg7nw\" (UniqueName: \"kubernetes.io/projected/e40f5ce9-777c-44d7-8325-6bb672235db7-kube-api-access-mg7nw\") pod \"e40f5ce9-777c-44d7-8325-6bb672235db7\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " Oct 30 09:57:52.770778 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.770729 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-credential-socket\") pod \"e40f5ce9-777c-44d7-8325-6bb672235db7\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " Oct 30 09:57:52.771076 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.770809 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-podinfo\") pod \"e40f5ce9-777c-44d7-8325-6bb672235db7\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " Oct 30 09:57:52.771076 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.770864 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-envoy\") pod \"e40f5ce9-777c-44d7-8325-6bb672235db7\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " Oct 30 09:57:52.771076 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.770944 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-workload-certs\") pod \"e40f5ce9-777c-44d7-8325-6bb672235db7\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " Oct 30 09:57:52.771076 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.771002 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-data\") pod \"e40f5ce9-777c-44d7-8325-6bb672235db7\" (UID: \"e40f5ce9-777c-44d7-8325-6bb672235db7\") " Oct 30 09:57:52.771724 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.771680 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-data" pod "e40f5ce9-777c-44d7-8325-6bb672235db7" (UID: "e40f5ce9-777c-44d7-8325-6bb672235db7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:52.771930 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.771898 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-credential-socket" pod "e40f5ce9-777c-44d7-8325-6bb672235db7" (UID: "e40f5ce9-777c-44d7-8325-6bb672235db7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:52.772072 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.772045 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-workload-certs" pod "e40f5ce9-777c-44d7-8325-6bb672235db7" (UID: "e40f5ce9-777c-44d7-8325-6bb672235db7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:52.772326 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.772304 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e40f5ce9-777c-44d7-8325-6bb672235db7-istiod-ca-cert" pod "e40f5ce9-777c-44d7-8325-6bb672235db7" (UID: "e40f5ce9-777c-44d7-8325-6bb672235db7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:52.772410 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.772352 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-workload-socket" pod "e40f5ce9-777c-44d7-8325-6bb672235db7" (UID: "e40f5ce9-777c-44d7-8325-6bb672235db7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:52.774500 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.774464 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-token" pod "e40f5ce9-777c-44d7-8325-6bb672235db7" (UID: "e40f5ce9-777c-44d7-8325-6bb672235db7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:52.774787 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.774761 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-envoy" pod "e40f5ce9-777c-44d7-8325-6bb672235db7" (UID: "e40f5ce9-777c-44d7-8325-6bb672235db7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:52.774787 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.774765 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-podinfo" pod "e40f5ce9-777c-44d7-8325-6bb672235db7" (UID: "e40f5ce9-777c-44d7-8325-6bb672235db7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:57:52.780233 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.780207 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40f5ce9-777c-44d7-8325-6bb672235db7-kube-api-access-mg7nw" pod "e40f5ce9-777c-44d7-8325-6bb672235db7" (UID: "e40f5ce9-777c-44d7-8325-6bb672235db7"). InnerVolumeSpecName "kube-api-access-mg7nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:52.872350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.872311 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg7nw\" (UniqueName: \"kubernetes.io/projected/e40f5ce9-777c-44d7-8325-6bb672235db7-kube-api-access-mg7nw\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:52.872350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.872346 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:52.872562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.872361 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:52.872562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.872373 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:52.872562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.872387 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:52.872562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.872400 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:52.872562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.872412 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40f5ce9-777c-44d7-8325-6bb672235db7-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:52.872562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.872422 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40f5ce9-777c-44d7-8325-6bb672235db7-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:52.872562 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.872436 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40f5ce9-777c-44d7-8325-6bb672235db7-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:57:52.995246 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.995152 5938 generic.go:356] "Generic (PLEG): container finished" podID="e40f5ce9-777c-44d7-8325-6bb672235db7" containerID="232d928ec1daf5303c9f5aa0908a2bad62550ae6110b7ae0275d0bfbc071fd99" exitCode=0 Oct 30 09:57:52.995246 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.995205 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" Oct 30 09:57:52.995246 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.995236 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" event={"ID":"e40f5ce9-777c-44d7-8325-6bb672235db7","Type":"ContainerDied","Data":"232d928ec1daf5303c9f5aa0908a2bad62550ae6110b7ae0275d0bfbc071fd99"} Oct 30 09:57:52.995555 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.995297 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm" event={"ID":"e40f5ce9-777c-44d7-8325-6bb672235db7","Type":"ContainerDied","Data":"5fba1bb871e635f0ae4e2b70119325d7a5fba06f980fe65395087006afeec42f"} Oct 30 09:57:52.995555 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:52.995327 5938 scope.go:122] "RemoveContainer" containerID="232d928ec1daf5303c9f5aa0908a2bad62550ae6110b7ae0275d0bfbc071fd99" Oct 30 09:57:53.007938 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:53.007902 5938 scope.go:122] "RemoveContainer" containerID="232d928ec1daf5303c9f5aa0908a2bad62550ae6110b7ae0275d0bfbc071fd99" Oct 30 09:57:53.008290 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:57:53.008268 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232d928ec1daf5303c9f5aa0908a2bad62550ae6110b7ae0275d0bfbc071fd99\": container with ID starting with 232d928ec1daf5303c9f5aa0908a2bad62550ae6110b7ae0275d0bfbc071fd99 not found: ID does not exist" containerID="232d928ec1daf5303c9f5aa0908a2bad62550ae6110b7ae0275d0bfbc071fd99" Oct 30 09:57:53.008404 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:53.008298 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232d928ec1daf5303c9f5aa0908a2bad62550ae6110b7ae0275d0bfbc071fd99"} err="failed to get container status \"232d928ec1daf5303c9f5aa0908a2bad62550ae6110b7ae0275d0bfbc071fd99\": rpc error: code = NotFound desc = could not find container \"232d928ec1daf5303c9f5aa0908a2bad62550ae6110b7ae0275d0bfbc071fd99\": container with ID starting with 232d928ec1daf5303c9f5aa0908a2bad62550ae6110b7ae0275d0bfbc071fd99 not found: ID does not exist" Oct 30 09:57:53.056700 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:53.056670 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm"] Oct 30 09:57:53.061561 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:53.061538 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ln7sm"] Oct 30 09:57:54.257330 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:57:54.257290 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e40f5ce9-777c-44d7-8325-6bb672235db7" path="/var/lib/kubelet/pods/e40f5ce9-777c-44d7-8325-6bb672235db7/volumes" Oct 30 09:58:07.023950 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:07.023910 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4"] Oct 30 09:58:07.057081 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:07.057050 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d"] Oct 30 09:58:08.027605 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:08.027558 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw"] Oct 30 09:58:08.041017 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:08.040992 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d0a328-xbbfw"] Oct 30 09:58:08.257076 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:08.257042 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8736557b-59cd-4e5e-8f14-73ac0cf641c5" path="/var/lib/kubelet/pods/8736557b-59cd-4e5e-8f14-73ac0cf641c5/volumes" Oct 30 09:58:09.026422 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:09.026384 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz"] Oct 30 09:58:09.039887 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:09.039860 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3e7b2b-5mvfz"] Oct 30 09:58:10.260388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:10.260350 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f9aa00-8f06-49b2-bb82-f749d7960915" path="/var/lib/kubelet/pods/c9f9aa00-8f06-49b2-bb82-f749d7960915/volumes" Oct 30 09:58:12.025140 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:12.025098 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" podUID="22d5adc2-3e45-4196-b59f-e2a585cb0714" containerName="payload-processing" containerID="cri-o://fe2ccea71296a9ae269d50f4c6867fd5a42d439030131e492d0f4e4ba32bf7df" gracePeriod=25 Oct 30 09:58:12.057632 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:12.057572 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" podUID="03aa9f18-97ff-48ed-9aa0-fcea685be2b2" containerName="payload-pre-processing" containerID="cri-o://d0a01eded410cc016a7287c31bd2e55ddbb143b29127488411d85c2ba4145555" gracePeriod=25 Oct 30 09:58:15.780609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:15.780548 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" podUID="22d5adc2-3e45-4196-b59f-e2a585cb0714" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:58:15.781099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:15.780610 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" podUID="03aa9f18-97ff-48ed-9aa0-fcea685be2b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:58:15.808487 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:58:15.808460 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:58:19.027190 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:19.027148 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b"] Oct 30 09:58:19.040370 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:19.040346 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39b8bd03-dnq2b"] Oct 30 09:58:20.027477 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:20.027437 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw"] Oct 30 09:58:20.040461 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:20.040432 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6e78aa-phqvw"] Oct 30 09:58:20.257641 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:20.257580 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="375796be-d502-4249-bf0a-9e32ab37aefa" path="/var/lib/kubelet/pods/375796be-d502-4249-bf0a-9e32ab37aefa/volumes" Oct 30 09:58:20.258363 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:20.258346 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb81527-7ae4-4642-91b8-26160d49ac55" path="/var/lib/kubelet/pods/5cb81527-7ae4-4642-91b8-26160d49ac55/volumes" Oct 30 09:58:23.122350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.122304 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd"] Oct 30 09:58:23.122866 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.122718 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" podUID="5a094c60-d6c5-4f30-8085-6d8381eb70b2" containerName="discovery" containerID="cri-o://1f024f1965b3cf41a3866bc9919b5e45d0734c4267a20c01fa0ff5a9e8b595ac" gracePeriod=30 Oct 30 09:58:23.397745 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.397717 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:58:23.541015 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.540968 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-cacerts\" (UniqueName: \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-cacerts\") pod \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " Oct 30 09:58:23.541015 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.541025 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-csr-ca-configmap\") pod \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " Oct 30 09:58:23.541269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.541090 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-kubeconfig\") pod \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " Oct 30 09:58:23.541269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.541192 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a094c60-d6c5-4f30-8085-6d8381eb70b2-kube-api-access-gcn2r\" (UniqueName: \"kubernetes.io/projected/5a094c60-d6c5-4f30-8085-6d8381eb70b2-kube-api-access-gcn2r\") pod \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " Oct 30 09:58:23.541269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.541236 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-token\" (UniqueName: \"kubernetes.io/projected/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-token\") pod \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " Oct 30 09:58:23.541443 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.541284 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-csr-dns-cert\") pod \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " Oct 30 09:58:23.541443 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.541362 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a094c60-d6c5-4f30-8085-6d8381eb70b2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a094c60-d6c5-4f30-8085-6d8381eb70b2-local-certs\") pod \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\" (UID: \"5a094c60-d6c5-4f30-8085-6d8381eb70b2\") " Oct 30 09:58:23.541550 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.541500 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-csr-ca-configmap" pod "5a094c60-d6c5-4f30-8085-6d8381eb70b2" (UID: "5a094c60-d6c5-4f30-8085-6d8381eb70b2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:23.542271 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.542233 5938 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-csr-ca-configmap\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:23.543895 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.543856 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-cacerts" pod "5a094c60-d6c5-4f30-8085-6d8381eb70b2" (UID: "5a094c60-d6c5-4f30-8085-6d8381eb70b2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:58:23.544198 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.544170 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-token" pod "5a094c60-d6c5-4f30-8085-6d8381eb70b2" (UID: "5a094c60-d6c5-4f30-8085-6d8381eb70b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:23.544346 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.544322 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a094c60-d6c5-4f30-8085-6d8381eb70b2-kube-api-access-gcn2r" pod "5a094c60-d6c5-4f30-8085-6d8381eb70b2" (UID: "5a094c60-d6c5-4f30-8085-6d8381eb70b2"). InnerVolumeSpecName "kube-api-access-gcn2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:23.544515 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.544499 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-csr-dns-cert" pod "5a094c60-d6c5-4f30-8085-6d8381eb70b2" (UID: "5a094c60-d6c5-4f30-8085-6d8381eb70b2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:58:23.544685 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.544666 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-kubeconfig" pod "5a094c60-d6c5-4f30-8085-6d8381eb70b2" (UID: "5a094c60-d6c5-4f30-8085-6d8381eb70b2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:58:23.544763 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.544684 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a094c60-d6c5-4f30-8085-6d8381eb70b2-local-certs" pod "5a094c60-d6c5-4f30-8085-6d8381eb70b2" (UID: "5a094c60-d6c5-4f30-8085-6d8381eb70b2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:23.644088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.644033 5938 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-kubeconfig\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:23.644088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.644057 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcn2r\" (UniqueName: \"kubernetes.io/projected/5a094c60-d6c5-4f30-8085-6d8381eb70b2-kube-api-access-gcn2r\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:23.644088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.644066 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:23.644088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.644076 5938 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-istio-csr-dns-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:23.644088 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.644087 5938 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a094c60-d6c5-4f30-8085-6d8381eb70b2-local-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:23.644339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:23.644095 5938 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5a094c60-d6c5-4f30-8085-6d8381eb70b2-cacerts\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:24.235475 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:24.235441 5938 generic.go:356] "Generic (PLEG): container finished" podID="5a094c60-d6c5-4f30-8085-6d8381eb70b2" containerID="1f024f1965b3cf41a3866bc9919b5e45d0734c4267a20c01fa0ff5a9e8b595ac" exitCode=0 Oct 30 09:58:24.236009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:24.235552 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" event={"ID":"5a094c60-d6c5-4f30-8085-6d8381eb70b2","Type":"ContainerDied","Data":"1f024f1965b3cf41a3866bc9919b5e45d0734c4267a20c01fa0ff5a9e8b595ac"} Oct 30 09:58:24.236009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:24.235614 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" Oct 30 09:58:24.236009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:24.235659 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd" event={"ID":"5a094c60-d6c5-4f30-8085-6d8381eb70b2","Type":"ContainerDied","Data":"c5eb8396c75c5a55d808cd65d5d10e3b869928f502d171b67ba13b7cee66017b"} Oct 30 09:58:24.236009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:24.235689 5938 scope.go:122] "RemoveContainer" containerID="1f024f1965b3cf41a3866bc9919b5e45d0734c4267a20c01fa0ff5a9e8b595ac" Oct 30 09:58:24.247238 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:24.247222 5938 scope.go:122] "RemoveContainer" containerID="1f024f1965b3cf41a3866bc9919b5e45d0734c4267a20c01fa0ff5a9e8b595ac" Oct 30 09:58:24.247510 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:58:24.247491 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f024f1965b3cf41a3866bc9919b5e45d0734c4267a20c01fa0ff5a9e8b595ac\": container with ID starting with 1f024f1965b3cf41a3866bc9919b5e45d0734c4267a20c01fa0ff5a9e8b595ac not found: ID does not exist" containerID="1f024f1965b3cf41a3866bc9919b5e45d0734c4267a20c01fa0ff5a9e8b595ac" Oct 30 09:58:24.247610 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:24.247517 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f024f1965b3cf41a3866bc9919b5e45d0734c4267a20c01fa0ff5a9e8b595ac"} err="failed to get container status \"1f024f1965b3cf41a3866bc9919b5e45d0734c4267a20c01fa0ff5a9e8b595ac\": rpc error: code = NotFound desc = could not find container \"1f024f1965b3cf41a3866bc9919b5e45d0734c4267a20c01fa0ff5a9e8b595ac\": container with ID starting with 1f024f1965b3cf41a3866bc9919b5e45d0734c4267a20c01fa0ff5a9e8b595ac not found: ID does not exist" Oct 30 09:58:24.304908 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:24.304886 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd"] Oct 30 09:58:24.318124 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:24.318100 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z6xd"] Oct 30 09:58:25.780530 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:25.780493 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" podUID="22d5adc2-3e45-4196-b59f-e2a585cb0714" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:58:25.781038 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:25.780566 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" podUID="03aa9f18-97ff-48ed-9aa0-fcea685be2b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:58:26.255983 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:26.255946 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a094c60-d6c5-4f30-8085-6d8381eb70b2" path="/var/lib/kubelet/pods/5a094c60-d6c5-4f30-8085-6d8381eb70b2/volumes" Oct 30 09:58:28.130869 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:28.130838 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc"] Oct 30 09:58:31.031353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:31.031317 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7"] Oct 30 09:58:31.044313 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:31.044279 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b0fd9d-jkgm7"] Oct 30 09:58:32.051002 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.050977 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg"] Oct 30 09:58:32.051424 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.051174 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" podUID="fc7bd19a-03fc-4246-b529-dd67becce442" containerName="maas-api" containerID="cri-o://63957b9fa187491c03319456f5862417dbfba0b0bb8c569b08e8c795c03f44d6" gracePeriod=30 Oct 30 09:58:32.275935 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.275638 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5faef7a2-55ae-4c8e-9787-43dfa5a1b67a" path="/var/lib/kubelet/pods/5faef7a2-55ae-4c8e-9787-43dfa5a1b67a/volumes" Oct 30 09:58:32.332141 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.330982 5938 generic.go:356] "Generic (PLEG): container finished" podID="03aa9f18-97ff-48ed-9aa0-fcea685be2b2" containerID="d0a01eded410cc016a7287c31bd2e55ddbb143b29127488411d85c2ba4145555" exitCode=0 Oct 30 09:58:32.332141 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.331095 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" event={"ID":"03aa9f18-97ff-48ed-9aa0-fcea685be2b2","Type":"ContainerDied","Data":"d0a01eded410cc016a7287c31bd2e55ddbb143b29127488411d85c2ba4145555"} Oct 30 09:58:32.334623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.333805 5938 generic.go:356] "Generic (PLEG): container finished" podID="fc7bd19a-03fc-4246-b529-dd67becce442" containerID="63957b9fa187491c03319456f5862417dbfba0b0bb8c569b08e8c795c03f44d6" exitCode=0 Oct 30 09:58:32.334623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.333957 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" event={"ID":"fc7bd19a-03fc-4246-b529-dd67becce442","Type":"ContainerDied","Data":"63957b9fa187491c03319456f5862417dbfba0b0bb8c569b08e8c795c03f44d6"} Oct 30 09:58:32.335942 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.335779 5938 generic.go:356] "Generic (PLEG): container finished" podID="22d5adc2-3e45-4196-b59f-e2a585cb0714" containerID="fe2ccea71296a9ae269d50f4c6867fd5a42d439030131e492d0f4e4ba32bf7df" exitCode=0 Oct 30 09:58:32.336052 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.335945 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" event={"ID":"22d5adc2-3e45-4196-b59f-e2a585cb0714","Type":"ContainerDied","Data":"fe2ccea71296a9ae269d50f4c6867fd5a42d439030131e492d0f4e4ba32bf7df"} Oct 30 09:58:32.357578 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.357556 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:58:32.455609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.448189 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7bd19a-03fc-4246-b529-dd67becce442-kube-api-access-x5659\" (UniqueName: \"kubernetes.io/projected/fc7bd19a-03fc-4246-b529-dd67becce442-kube-api-access-x5659\") pod \"fc7bd19a-03fc-4246-b529-dd67becce442\" (UID: \"fc7bd19a-03fc-4246-b529-dd67becce442\") " Oct 30 09:58:32.455609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.448268 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc7bd19a-03fc-4246-b529-dd67becce442-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc7bd19a-03fc-4246-b529-dd67becce442-openshift-service-ca\") pod \"fc7bd19a-03fc-4246-b529-dd67becce442\" (UID: \"fc7bd19a-03fc-4246-b529-dd67becce442\") " Oct 30 09:58:32.455609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.448322 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc7bd19a-03fc-4246-b529-dd67becce442-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc7bd19a-03fc-4246-b529-dd67becce442-maas-api-tls\") pod \"fc7bd19a-03fc-4246-b529-dd67becce442\" (UID: \"fc7bd19a-03fc-4246-b529-dd67becce442\") " Oct 30 09:58:32.455609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.448379 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc7bd19a-03fc-4246-b529-dd67becce442-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7bd19a-03fc-4246-b529-dd67becce442-metrics-certs\") pod \"fc7bd19a-03fc-4246-b529-dd67becce442\" (UID: \"fc7bd19a-03fc-4246-b529-dd67becce442\") " Oct 30 09:58:32.455609 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.449757 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc7bd19a-03fc-4246-b529-dd67becce442-openshift-service-ca" pod "fc7bd19a-03fc-4246-b529-dd67becce442" (UID: "fc7bd19a-03fc-4246-b529-dd67becce442"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:32.462775 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.459892 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc7bd19a-03fc-4246-b529-dd67becce442-maas-api-tls" pod "fc7bd19a-03fc-4246-b529-dd67becce442" (UID: "fc7bd19a-03fc-4246-b529-dd67becce442"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:58:32.462775 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.459978 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7bd19a-03fc-4246-b529-dd67becce442-kube-api-access-x5659" pod "fc7bd19a-03fc-4246-b529-dd67becce442" (UID: "fc7bd19a-03fc-4246-b529-dd67becce442"). InnerVolumeSpecName "kube-api-access-x5659". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:32.462775 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.460104 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc7bd19a-03fc-4246-b529-dd67becce442-metrics-certs" pod "fc7bd19a-03fc-4246-b529-dd67becce442" (UID: "fc7bd19a-03fc-4246-b529-dd67becce442"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:58:32.464956 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.464932 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" Oct 30 09:58:32.469834 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.469815 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" Oct 30 09:58:32.550556 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.550469 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5659\" (UniqueName: \"kubernetes.io/projected/fc7bd19a-03fc-4246-b529-dd67becce442-kube-api-access-x5659\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:32.550556 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.550505 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc7bd19a-03fc-4246-b529-dd67becce442-openshift-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:32.550556 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.550522 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc7bd19a-03fc-4246-b529-dd67becce442-maas-api-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:32.550556 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.550535 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7bd19a-03fc-4246-b529-dd67becce442-metrics-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:32.651626 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.651572 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d5adc2-3e45-4196-b59f-e2a585cb0714-kube-api-access-sfr62\" (UniqueName: \"kubernetes.io/projected/22d5adc2-3e45-4196-b59f-e2a585cb0714-kube-api-access-sfr62\") pod \"22d5adc2-3e45-4196-b59f-e2a585cb0714\" (UID: \"22d5adc2-3e45-4196-b59f-e2a585cb0714\") " Oct 30 09:58:32.651790 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.651644 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03aa9f18-97ff-48ed-9aa0-fcea685be2b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03aa9f18-97ff-48ed-9aa0-fcea685be2b2-plugins-config-volume\") pod \"03aa9f18-97ff-48ed-9aa0-fcea685be2b2\" (UID: \"03aa9f18-97ff-48ed-9aa0-fcea685be2b2\") " Oct 30 09:58:32.651790 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.651731 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22d5adc2-3e45-4196-b59f-e2a585cb0714-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d5adc2-3e45-4196-b59f-e2a585cb0714-plugins-config-volume\") pod \"22d5adc2-3e45-4196-b59f-e2a585cb0714\" (UID: \"22d5adc2-3e45-4196-b59f-e2a585cb0714\") " Oct 30 09:58:32.651790 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.651784 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03aa9f18-97ff-48ed-9aa0-fcea685be2b2-kube-api-access-rrwh8\" (UniqueName: \"kubernetes.io/projected/03aa9f18-97ff-48ed-9aa0-fcea685be2b2-kube-api-access-rrwh8\") pod \"03aa9f18-97ff-48ed-9aa0-fcea685be2b2\" (UID: \"03aa9f18-97ff-48ed-9aa0-fcea685be2b2\") " Oct 30 09:58:32.652098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.652075 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03aa9f18-97ff-48ed-9aa0-fcea685be2b2-plugins-config-volume" pod "03aa9f18-97ff-48ed-9aa0-fcea685be2b2" (UID: "03aa9f18-97ff-48ed-9aa0-fcea685be2b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:32.652181 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.652078 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d5adc2-3e45-4196-b59f-e2a585cb0714-plugins-config-volume" pod "22d5adc2-3e45-4196-b59f-e2a585cb0714" (UID: "22d5adc2-3e45-4196-b59f-e2a585cb0714"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:32.652254 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.652232 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03aa9f18-97ff-48ed-9aa0-fcea685be2b2-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:32.652311 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.652264 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d5adc2-3e45-4196-b59f-e2a585cb0714-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:32.653940 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.653912 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d5adc2-3e45-4196-b59f-e2a585cb0714-kube-api-access-sfr62" pod "22d5adc2-3e45-4196-b59f-e2a585cb0714" (UID: "22d5adc2-3e45-4196-b59f-e2a585cb0714"). InnerVolumeSpecName "kube-api-access-sfr62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:32.653940 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.653930 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03aa9f18-97ff-48ed-9aa0-fcea685be2b2-kube-api-access-rrwh8" pod "03aa9f18-97ff-48ed-9aa0-fcea685be2b2" (UID: "03aa9f18-97ff-48ed-9aa0-fcea685be2b2"). InnerVolumeSpecName "kube-api-access-rrwh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:32.753447 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.753420 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfr62\" (UniqueName: \"kubernetes.io/projected/22d5adc2-3e45-4196-b59f-e2a585cb0714-kube-api-access-sfr62\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:32.753447 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:32.753442 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrwh8\" (UniqueName: \"kubernetes.io/projected/03aa9f18-97ff-48ed-9aa0-fcea685be2b2-kube-api-access-rrwh8\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:33.029460 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.029428 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888"] Oct 30 09:58:33.031758 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.031727 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb070777-g4888"] Oct 30 09:58:33.132191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.132131 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" podUID="2089f287-3340-45e9-99d4-9e3ce1fbbe9e" containerName="payload-processing" containerID="cri-o://4cf35131c08463da4e5264099fd47802b7e80952a9fbcc0621ddc7738a5b6425" gracePeriod=25 Oct 30 09:58:33.344859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.344775 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" Oct 30 09:58:33.344859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.344816 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg" event={"ID":"fc7bd19a-03fc-4246-b529-dd67becce442","Type":"ContainerDied","Data":"c762f2d0d80b31d3b712e4fbb6f1f620bba340681f66f20f3443727192a93819"} Oct 30 09:58:33.345105 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.344886 5938 scope.go:122] "RemoveContainer" containerID="63957b9fa187491c03319456f5862417dbfba0b0bb8c569b08e8c795c03f44d6" Oct 30 09:58:33.346254 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.346233 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" Oct 30 09:58:33.346378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.346279 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4" event={"ID":"22d5adc2-3e45-4196-b59f-e2a585cb0714","Type":"ContainerDied","Data":"3c43d00aa4fd2c003e0396e3ad08d017c0afc5298cf3157afd7f40bd2af2d644"} Oct 30 09:58:33.347927 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.347907 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" event={"ID":"03aa9f18-97ff-48ed-9aa0-fcea685be2b2","Type":"ContainerDied","Data":"f0a314550d1b96699a376453a52b44ef74964370f219ac12a520a9860435d5ff"} Oct 30 09:58:33.348029 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.347939 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d" Oct 30 09:58:33.358554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.358533 5938 scope.go:122] "RemoveContainer" containerID="fe2ccea71296a9ae269d50f4c6867fd5a42d439030131e492d0f4e4ba32bf7df" Oct 30 09:58:33.370385 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.370368 5938 scope.go:122] "RemoveContainer" containerID="d0a01eded410cc016a7287c31bd2e55ddbb143b29127488411d85c2ba4145555" Oct 30 09:58:33.400891 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.400865 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d"] Oct 30 09:58:33.408623 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.408227 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ae795b-c4c647b59-btl9d"] Oct 30 09:58:33.439042 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.439018 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg"] Oct 30 09:58:33.441299 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.441279 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f6f96-78d65f4469-qrtrg"] Oct 30 09:58:33.475440 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.475417 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4"] Oct 30 09:58:33.480434 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:33.480412 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae795b-7b56f85c86-h8qt4"] Oct 30 09:58:34.257235 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:34.257203 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03aa9f18-97ff-48ed-9aa0-fcea685be2b2" path="/var/lib/kubelet/pods/03aa9f18-97ff-48ed-9aa0-fcea685be2b2/volumes" Oct 30 09:58:34.257631 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:34.257607 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22d5adc2-3e45-4196-b59f-e2a585cb0714" path="/var/lib/kubelet/pods/22d5adc2-3e45-4196-b59f-e2a585cb0714/volumes" Oct 30 09:58:34.257985 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:34.257970 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ab54ca-24c3-4bd3-a464-0d1a23bee04a" path="/var/lib/kubelet/pods/52ab54ca-24c3-4bd3-a464-0d1a23bee04a/volumes" Oct 30 09:58:34.258450 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:34.258432 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc7bd19a-03fc-4246-b529-dd67becce442" path="/var/lib/kubelet/pods/fc7bd19a-03fc-4246-b529-dd67becce442/volumes" Oct 30 09:58:37.199128 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.199096 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4"] Oct 30 09:58:37.199826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.199805 5938 cpu_manager.go:414] "containerMap: removing container" podUID="fc7bd19a-03fc-4246-b529-dd67becce442" containerName="maas-api" Oct 30 09:58:37.199826 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.199828 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc7bd19a-03fc-4246-b529-dd67becce442" containerName="maas-api" Oct 30 09:58:37.200009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.199852 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" containerName="istio-proxy" Oct 30 09:58:37.200009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.199860 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" containerName="istio-proxy" Oct 30 09:58:37.200009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.199876 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8621f4c3-0a60-4fdb-ab7a-f520bfc3e659" containerName="revoke-keys" Oct 30 09:58:37.200009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.199885 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8621f4c3-0a60-4fdb-ab7a-f520bfc3e659" containerName="revoke-keys" Oct 30 09:58:37.200009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.199901 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerName="main" Oct 30 09:58:37.200009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.199908 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerName="main" Oct 30 09:58:37.200009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.199930 5938 cpu_manager.go:414] "containerMap: removing container" podUID="03aa9f18-97ff-48ed-9aa0-fcea685be2b2" containerName="payload-pre-processing" Oct 30 09:58:37.200009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.199939 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03aa9f18-97ff-48ed-9aa0-fcea685be2b2" containerName="payload-pre-processing" Oct 30 09:58:37.200009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.199956 5938 cpu_manager.go:414] "containerMap: removing container" podUID="22d5adc2-3e45-4196-b59f-e2a585cb0714" containerName="payload-processing" Oct 30 09:58:37.200009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.199964 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d5adc2-3e45-4196-b59f-e2a585cb0714" containerName="payload-processing" Oct 30 09:58:37.200009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.199983 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" Oct 30 09:58:37.200009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.199990 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" Oct 30 09:58:37.200009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.200000 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef" containerName="revoke-keys" Oct 30 09:58:37.200009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.200007 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef" containerName="revoke-keys" Oct 30 09:58:37.200747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.200021 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5a094c60-d6c5-4f30-8085-6d8381eb70b2" containerName="discovery" Oct 30 09:58:37.200747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.200028 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a094c60-d6c5-4f30-8085-6d8381eb70b2" containerName="discovery" Oct 30 09:58:37.200747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.200042 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e40f5ce9-777c-44d7-8325-6bb672235db7" containerName="istio-proxy" Oct 30 09:58:37.200747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.200049 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e40f5ce9-777c-44d7-8325-6bb672235db7" containerName="istio-proxy" Oct 30 09:58:37.200747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.200174 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ded0c49-3011-4730-af91-f1700a3034b3" containerName="main" Oct 30 09:58:37.200747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.200189 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef" containerName="revoke-keys" Oct 30 09:58:37.200747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.200200 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a094c60-d6c5-4f30-8085-6d8381eb70b2" containerName="discovery" Oct 30 09:58:37.200747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.200213 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e40f5ce9-777c-44d7-8325-6bb672235db7" containerName="istio-proxy" Oct 30 09:58:37.200747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.200227 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc7bd19a-03fc-4246-b529-dd67becce442" containerName="maas-api" Oct 30 09:58:37.200747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.200237 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6de3fa7-ddd5-4816-b6b9-f18bae20bf0d" containerName="istio-proxy" Oct 30 09:58:37.200747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.200247 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8621f4c3-0a60-4fdb-ab7a-f520bfc3e659" containerName="revoke-keys" Oct 30 09:58:37.200747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.200256 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f900f402-3458-4a8e-aa3b-a92b7ede222f" containerName="main" Oct 30 09:58:37.200747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.200267 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="03aa9f18-97ff-48ed-9aa0-fcea685be2b2" containerName="payload-pre-processing" Oct 30 09:58:37.200747 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.200277 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d5adc2-3e45-4196-b59f-e2a585cb0714" containerName="payload-processing" Oct 30 09:58:37.207905 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.207882 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" Oct 30 09:58:37.220835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.220807 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4"] Oct 30 09:58:37.231657 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.231632 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:58:37.231785 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.231728 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:58:37.303467 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.303440 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szvds\" (UniqueName: \"kubernetes.io/projected/3f56fdd1-d21a-4007-bf18-1d76cb519e65-kube-api-access-szvds\") pod \"maas-api-revoke-sub-284af7d5-kngd4\" (UID: \"3f56fdd1-d21a-4007-bf18-1d76cb519e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" Oct 30 09:58:37.303655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.303509 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f56fdd1-d21a-4007-bf18-1d76cb519e65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-284af7d5-kngd4\" (UID: \"3f56fdd1-d21a-4007-bf18-1d76cb519e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" Oct 30 09:58:37.303748 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.303681 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f56fdd1-d21a-4007-bf18-1d76cb519e65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-284af7d5-kngd4\" (UID: \"3f56fdd1-d21a-4007-bf18-1d76cb519e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" Oct 30 09:58:37.404489 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.404459 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f56fdd1-d21a-4007-bf18-1d76cb519e65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-284af7d5-kngd4\" (UID: \"3f56fdd1-d21a-4007-bf18-1d76cb519e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" Oct 30 09:58:37.404667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.404516 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f56fdd1-d21a-4007-bf18-1d76cb519e65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-284af7d5-kngd4\" (UID: \"3f56fdd1-d21a-4007-bf18-1d76cb519e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" Oct 30 09:58:37.404667 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.404648 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szvds\" (UniqueName: \"kubernetes.io/projected/3f56fdd1-d21a-4007-bf18-1d76cb519e65-kube-api-access-szvds\") pod \"maas-api-revoke-sub-284af7d5-kngd4\" (UID: \"3f56fdd1-d21a-4007-bf18-1d76cb519e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" Oct 30 09:58:37.405255 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.405237 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f56fdd1-d21a-4007-bf18-1d76cb519e65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-284af7d5-kngd4\" (UID: \"3f56fdd1-d21a-4007-bf18-1d76cb519e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" Oct 30 09:58:37.407133 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.407111 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f56fdd1-d21a-4007-bf18-1d76cb519e65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-284af7d5-kngd4\" (UID: \"3f56fdd1-d21a-4007-bf18-1d76cb519e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" Oct 30 09:58:37.425352 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.425326 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szvds\" (UniqueName: \"kubernetes.io/projected/3f56fdd1-d21a-4007-bf18-1d76cb519e65-kube-api-access-szvds\") pod \"maas-api-revoke-sub-284af7d5-kngd4\" (UID: \"3f56fdd1-d21a-4007-bf18-1d76cb519e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" Oct 30 09:58:37.521482 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.521420 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" Oct 30 09:58:37.821373 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:37.821349 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4"] Oct 30 09:58:37.823025 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:58:37.822997 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f56fdd1_d21a_4007_bf18_1d76cb519e65.slice/crio-991c4f949d998e70514589673db31b14e68d4699f6d05f0d7376e0392b524c27 WatchSource:0}: Error finding container 991c4f949d998e70514589673db31b14e68d4699f6d05f0d7376e0392b524c27: Status 404 returned error can't find the container with id 991c4f949d998e70514589673db31b14e68d4699f6d05f0d7376e0392b524c27 Oct 30 09:58:38.320083 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:38.320046 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" podUID="2089f287-3340-45e9-99d4-9e3ce1fbbe9e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:58:38.388080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:38.387992 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" event={"ID":"3f56fdd1-d21a-4007-bf18-1d76cb519e65","Type":"ContainerStarted","Data":"9c73d19bff9ebdec1e27bc3513e62a203a5f777c3e0ef012dbd13fc32ba7eeac"} Oct 30 09:58:38.388080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:38.388053 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" event={"ID":"3f56fdd1-d21a-4007-bf18-1d76cb519e65","Type":"ContainerStarted","Data":"991c4f949d998e70514589673db31b14e68d4699f6d05f0d7376e0392b524c27"} Oct 30 09:58:38.429299 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:38.429240 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" podStartSLOduration=1.429228332 podStartE2EDuration="1.429228332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:58:38.428293157 +0000 UTC m=+3836.290998979" watchObservedRunningTime="2026-10-30 09:58:38.429228332 +0000 UTC m=+3836.291934153" Oct 30 09:58:39.397110 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:39.397078 5938 generic.go:356] "Generic (PLEG): container finished" podID="3f56fdd1-d21a-4007-bf18-1d76cb519e65" containerID="9c73d19bff9ebdec1e27bc3513e62a203a5f777c3e0ef012dbd13fc32ba7eeac" exitCode=0 Oct 30 09:58:39.397543 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:39.397173 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" event={"ID":"3f56fdd1-d21a-4007-bf18-1d76cb519e65","Type":"ContainerDied","Data":"9c73d19bff9ebdec1e27bc3513e62a203a5f777c3e0ef012dbd13fc32ba7eeac"} Oct 30 09:58:40.551841 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.551818 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" Oct 30 09:58:40.640223 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.640196 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f56fdd1-d21a-4007-bf18-1d76cb519e65-kube-api-access-szvds\" (UniqueName: \"kubernetes.io/projected/3f56fdd1-d21a-4007-bf18-1d76cb519e65-kube-api-access-szvds\") pod \"3f56fdd1-d21a-4007-bf18-1d76cb519e65\" (UID: \"3f56fdd1-d21a-4007-bf18-1d76cb519e65\") " Oct 30 09:58:40.640391 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.640296 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f56fdd1-d21a-4007-bf18-1d76cb519e65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f56fdd1-d21a-4007-bf18-1d76cb519e65-maas-api-xks-ca\") pod \"3f56fdd1-d21a-4007-bf18-1d76cb519e65\" (UID: \"3f56fdd1-d21a-4007-bf18-1d76cb519e65\") " Oct 30 09:58:40.640460 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.640423 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f56fdd1-d21a-4007-bf18-1d76cb519e65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f56fdd1-d21a-4007-bf18-1d76cb519e65-maas-api-openshift-ca\") pod \"3f56fdd1-d21a-4007-bf18-1d76cb519e65\" (UID: \"3f56fdd1-d21a-4007-bf18-1d76cb519e65\") " Oct 30 09:58:40.640819 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.640792 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f56fdd1-d21a-4007-bf18-1d76cb519e65-maas-api-openshift-ca" pod "3f56fdd1-d21a-4007-bf18-1d76cb519e65" (UID: "3f56fdd1-d21a-4007-bf18-1d76cb519e65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:40.642470 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.642451 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f56fdd1-d21a-4007-bf18-1d76cb519e65-maas-api-xks-ca" pod "3f56fdd1-d21a-4007-bf18-1d76cb519e65" (UID: "3f56fdd1-d21a-4007-bf18-1d76cb519e65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:58:40.642633 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.642615 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f56fdd1-d21a-4007-bf18-1d76cb519e65-kube-api-access-szvds" pod "3f56fdd1-d21a-4007-bf18-1d76cb519e65" (UID: "3f56fdd1-d21a-4007-bf18-1d76cb519e65"). InnerVolumeSpecName "kube-api-access-szvds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:40.741758 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.741691 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szvds\" (UniqueName: \"kubernetes.io/projected/3f56fdd1-d21a-4007-bf18-1d76cb519e65-kube-api-access-szvds\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:40.741758 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.741718 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f56fdd1-d21a-4007-bf18-1d76cb519e65-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:40.741758 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.741731 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f56fdd1-d21a-4007-bf18-1d76cb519e65-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:40.840455 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.840416 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll"] Oct 30 09:58:40.840972 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.840959 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3f56fdd1-d21a-4007-bf18-1d76cb519e65" containerName="revoke-keys" Oct 30 09:58:40.841032 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.840974 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f56fdd1-d21a-4007-bf18-1d76cb519e65" containerName="revoke-keys" Oct 30 09:58:40.841080 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.841070 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f56fdd1-d21a-4007-bf18-1d76cb519e65" containerName="revoke-keys" Oct 30 09:58:40.844600 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.844568 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll" Oct 30 09:58:40.861565 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.861547 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll"] Oct 30 09:58:40.943901 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.943869 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e347845-b234-44ec-9a56-fe279c0913ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll\" (UID: \"9e347845-b234-44ec-9a56-fe279c0913ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll" Oct 30 09:58:40.944033 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.943909 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf8hq\" (UniqueName: \"kubernetes.io/projected/9e347845-b234-44ec-9a56-fe279c0913ba-kube-api-access-bf8hq\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll\" (UID: \"9e347845-b234-44ec-9a56-fe279c0913ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll" Oct 30 09:58:40.944095 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:40.944052 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e347845-b234-44ec-9a56-fe279c0913ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll\" (UID: \"9e347845-b234-44ec-9a56-fe279c0913ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll" Oct 30 09:58:41.045229 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:41.045162 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e347845-b234-44ec-9a56-fe279c0913ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll\" (UID: \"9e347845-b234-44ec-9a56-fe279c0913ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll" Oct 30 09:58:41.045360 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:41.045257 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e347845-b234-44ec-9a56-fe279c0913ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll\" (UID: \"9e347845-b234-44ec-9a56-fe279c0913ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll" Oct 30 09:58:41.045360 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:41.045284 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf8hq\" (UniqueName: \"kubernetes.io/projected/9e347845-b234-44ec-9a56-fe279c0913ba-kube-api-access-bf8hq\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll\" (UID: \"9e347845-b234-44ec-9a56-fe279c0913ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll" Oct 30 09:58:41.045888 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:41.045864 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e347845-b234-44ec-9a56-fe279c0913ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll\" (UID: \"9e347845-b234-44ec-9a56-fe279c0913ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll" Oct 30 09:58:41.047877 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:41.047853 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e347845-b234-44ec-9a56-fe279c0913ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll\" (UID: \"9e347845-b234-44ec-9a56-fe279c0913ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll" Oct 30 09:58:41.065795 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:41.065774 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf8hq\" (UniqueName: \"kubernetes.io/projected/9e347845-b234-44ec-9a56-fe279c0913ba-kube-api-access-bf8hq\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll\" (UID: \"9e347845-b234-44ec-9a56-fe279c0913ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll" Oct 30 09:58:41.158550 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:41.158527 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll" Oct 30 09:58:41.414523 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:41.414419 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" Oct 30 09:58:41.414523 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:41.414441 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284af7d5-kngd4" event={"ID":"3f56fdd1-d21a-4007-bf18-1d76cb519e65","Type":"ContainerDied","Data":"991c4f949d998e70514589673db31b14e68d4699f6d05f0d7376e0392b524c27"} Oct 30 09:58:41.414523 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:41.414496 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="991c4f949d998e70514589673db31b14e68d4699f6d05f0d7376e0392b524c27" Oct 30 09:58:41.459204 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:41.459098 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll"] Oct 30 09:58:41.460549 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:58:41.460523 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e347845_b234_44ec_9a56_fe279c0913ba.slice/crio-5c65761426cd365d2f268d55b321b41fda2ee3fb10340397d38933b0ce896e02 WatchSource:0}: Error finding container 5c65761426cd365d2f268d55b321b41fda2ee3fb10340397d38933b0ce896e02: Status 404 returned error can't find the container with id 5c65761426cd365d2f268d55b321b41fda2ee3fb10340397d38933b0ce896e02 Oct 30 09:58:42.033200 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:42.033119 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg"] Oct 30 09:58:42.046693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:42.046671 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd0e6e2-kzxhg"] Oct 30 09:58:42.265542 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:42.265498 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6102d5ea-7a81-45a0-81b0-afead85c8871" path="/var/lib/kubelet/pods/6102d5ea-7a81-45a0-81b0-afead85c8871/volumes" Oct 30 09:58:42.423197 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:42.423168 5938 generic.go:356] "Generic (PLEG): container finished" podID="9e347845-b234-44ec-9a56-fe279c0913ba" containerID="a188c1f552b15f28716c56057dc92875e0e199fe7d90d7caf7a0f8b55eb500b2" exitCode=0 Oct 30 09:58:42.423378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:42.423252 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll" event={"ID":"9e347845-b234-44ec-9a56-fe279c0913ba","Type":"ContainerDied","Data":"a188c1f552b15f28716c56057dc92875e0e199fe7d90d7caf7a0f8b55eb500b2"} Oct 30 09:58:42.423378 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:42.423313 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll" event={"ID":"9e347845-b234-44ec-9a56-fe279c0913ba","Type":"ContainerStarted","Data":"5c65761426cd365d2f268d55b321b41fda2ee3fb10340397d38933b0ce896e02"} Oct 30 09:58:43.577556 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:43.577533 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll" Oct 30 09:58:43.672796 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:43.672773 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e347845-b234-44ec-9a56-fe279c0913ba-kube-api-access-bf8hq\" (UniqueName: \"kubernetes.io/projected/9e347845-b234-44ec-9a56-fe279c0913ba-kube-api-access-bf8hq\") pod \"9e347845-b234-44ec-9a56-fe279c0913ba\" (UID: \"9e347845-b234-44ec-9a56-fe279c0913ba\") " Oct 30 09:58:43.672943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:43.672854 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e347845-b234-44ec-9a56-fe279c0913ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e347845-b234-44ec-9a56-fe279c0913ba-maas-api-xks-ca\") pod \"9e347845-b234-44ec-9a56-fe279c0913ba\" (UID: \"9e347845-b234-44ec-9a56-fe279c0913ba\") " Oct 30 09:58:43.672943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:43.672905 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e347845-b234-44ec-9a56-fe279c0913ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e347845-b234-44ec-9a56-fe279c0913ba-maas-api-openshift-ca\") pod \"9e347845-b234-44ec-9a56-fe279c0913ba\" (UID: \"9e347845-b234-44ec-9a56-fe279c0913ba\") " Oct 30 09:58:43.673311 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:43.673289 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e347845-b234-44ec-9a56-fe279c0913ba-maas-api-openshift-ca" pod "9e347845-b234-44ec-9a56-fe279c0913ba" (UID: "9e347845-b234-44ec-9a56-fe279c0913ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:43.674959 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:43.674931 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e347845-b234-44ec-9a56-fe279c0913ba-kube-api-access-bf8hq" pod "9e347845-b234-44ec-9a56-fe279c0913ba" (UID: "9e347845-b234-44ec-9a56-fe279c0913ba"). InnerVolumeSpecName "kube-api-access-bf8hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:43.675098 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:43.675016 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e347845-b234-44ec-9a56-fe279c0913ba-maas-api-xks-ca" pod "9e347845-b234-44ec-9a56-fe279c0913ba" (UID: "9e347845-b234-44ec-9a56-fe279c0913ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:58:43.774388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:43.774361 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf8hq\" (UniqueName: \"kubernetes.io/projected/9e347845-b234-44ec-9a56-fe279c0913ba-kube-api-access-bf8hq\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:43.774388 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:43.774385 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e347845-b234-44ec-9a56-fe279c0913ba-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:43.774547 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:43.774394 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e347845-b234-44ec-9a56-fe279c0913ba-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:44.026720 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:44.026624 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f"] Oct 30 09:58:44.028791 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:44.028765 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-891b53c6-qbk8f"] Oct 30 09:58:44.257336 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:44.257304 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4dacfa7-b373-403f-8881-537132dd2997" path="/var/lib/kubelet/pods/b4dacfa7-b373-403f-8881-537132dd2997/volumes" Oct 30 09:58:44.438350 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:44.438322 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll" Oct 30 09:58:44.438526 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:44.438344 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll" event={"ID":"9e347845-b234-44ec-9a56-fe279c0913ba","Type":"ContainerDied","Data":"5c65761426cd365d2f268d55b321b41fda2ee3fb10340397d38933b0ce896e02"} Oct 30 09:58:44.438526 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:44.438394 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c65761426cd365d2f268d55b321b41fda2ee3fb10340397d38933b0ce896e02" Oct 30 09:58:45.808420 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:58:45.808386 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:58:48.002176 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.002140 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt"] Oct 30 09:58:48.262397 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.262323 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj"] Oct 30 09:58:48.262986 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.262966 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9e347845-b234-44ec-9a56-fe279c0913ba" containerName="revoke-keys" Oct 30 09:58:48.262986 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.262986 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e347845-b234-44ec-9a56-fe279c0913ba" containerName="revoke-keys" Oct 30 09:58:48.263148 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.263115 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e347845-b234-44ec-9a56-fe279c0913ba" containerName="revoke-keys" Oct 30 09:58:48.266364 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.266344 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" Oct 30 09:58:48.288154 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.288120 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:58:48.288278 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.288218 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:58:48.291534 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.291513 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj"] Oct 30 09:58:48.320621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.320551 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" podUID="2089f287-3340-45e9-99d4-9e3ce1fbbe9e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:58:48.324226 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.324185 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pfhv\" (UniqueName: \"kubernetes.io/projected/dafca48f-8187-41c5-b23f-c9b600701701-kube-api-access-7pfhv\") pod \"maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj\" (UID: \"dafca48f-8187-41c5-b23f-c9b600701701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" Oct 30 09:58:48.324375 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.324291 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafca48f-8187-41c5-b23f-c9b600701701-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj\" (UID: \"dafca48f-8187-41c5-b23f-c9b600701701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" Oct 30 09:58:48.324457 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.324441 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafca48f-8187-41c5-b23f-c9b600701701-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj\" (UID: \"dafca48f-8187-41c5-b23f-c9b600701701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" Oct 30 09:58:48.425554 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.425521 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafca48f-8187-41c5-b23f-c9b600701701-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj\" (UID: \"dafca48f-8187-41c5-b23f-c9b600701701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" Oct 30 09:58:48.425745 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.425573 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pfhv\" (UniqueName: \"kubernetes.io/projected/dafca48f-8187-41c5-b23f-c9b600701701-kube-api-access-7pfhv\") pod \"maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj\" (UID: \"dafca48f-8187-41c5-b23f-c9b600701701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" Oct 30 09:58:48.425745 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.425645 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafca48f-8187-41c5-b23f-c9b600701701-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj\" (UID: \"dafca48f-8187-41c5-b23f-c9b600701701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" Oct 30 09:58:48.426492 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.426469 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafca48f-8187-41c5-b23f-c9b600701701-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj\" (UID: \"dafca48f-8187-41c5-b23f-c9b600701701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" Oct 30 09:58:48.430423 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.430400 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafca48f-8187-41c5-b23f-c9b600701701-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj\" (UID: \"dafca48f-8187-41c5-b23f-c9b600701701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" Oct 30 09:58:48.445916 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.445893 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pfhv\" (UniqueName: \"kubernetes.io/projected/dafca48f-8187-41c5-b23f-c9b600701701-kube-api-access-7pfhv\") pod \"maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj\" (UID: \"dafca48f-8187-41c5-b23f-c9b600701701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" Oct 30 09:58:48.579788 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.579701 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" Oct 30 09:58:48.881220 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:48.881189 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj"] Oct 30 09:58:48.882823 ip-10-0-135-148 kubenswrapper[5938]: W1030 09:58:48.882794 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafca48f_8187_41c5_b23f_c9b600701701.slice/crio-d75184b294dc853a8b53ffff865d489ce71d332c0c85c46a12a64cd2a9578696 WatchSource:0}: Error finding container d75184b294dc853a8b53ffff865d489ce71d332c0c85c46a12a64cd2a9578696: Status 404 returned error can't find the container with id d75184b294dc853a8b53ffff865d489ce71d332c0c85c46a12a64cd2a9578696 Oct 30 09:58:49.478960 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:49.478878 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" event={"ID":"dafca48f-8187-41c5-b23f-c9b600701701","Type":"ContainerStarted","Data":"7e8cc87d4d628e5169348c45dc29041721c34f0086f1f66caf962f26e7aef8c4"} Oct 30 09:58:49.478960 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:49.478957 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" event={"ID":"dafca48f-8187-41c5-b23f-c9b600701701","Type":"ContainerStarted","Data":"d75184b294dc853a8b53ffff865d489ce71d332c0c85c46a12a64cd2a9578696"} Oct 30 09:58:49.518917 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:49.518864 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" podStartSLOduration=1.518846896 podStartE2EDuration="1.518846896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:58:49.517421398 +0000 UTC m=+3847.380127219" watchObservedRunningTime="2026-10-30 09:58:49.518846896 +0000 UTC m=+3847.381552718" Oct 30 09:58:50.487099 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:50.487068 5938 generic.go:356] "Generic (PLEG): container finished" podID="dafca48f-8187-41c5-b23f-c9b600701701" containerID="7e8cc87d4d628e5169348c45dc29041721c34f0086f1f66caf962f26e7aef8c4" exitCode=0 Oct 30 09:58:50.487525 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:50.487167 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" event={"ID":"dafca48f-8187-41c5-b23f-c9b600701701","Type":"ContainerDied","Data":"7e8cc87d4d628e5169348c45dc29041721c34f0086f1f66caf962f26e7aef8c4"} Oct 30 09:58:51.651668 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:51.651646 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" Oct 30 09:58:51.765009 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:51.764928 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dafca48f-8187-41c5-b23f-c9b600701701-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafca48f-8187-41c5-b23f-c9b600701701-maas-api-xks-ca\") pod \"dafca48f-8187-41c5-b23f-c9b600701701\" (UID: \"dafca48f-8187-41c5-b23f-c9b600701701\") " Oct 30 09:58:51.765180 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:51.765025 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafca48f-8187-41c5-b23f-c9b600701701-kube-api-access-7pfhv\" (UniqueName: \"kubernetes.io/projected/dafca48f-8187-41c5-b23f-c9b600701701-kube-api-access-7pfhv\") pod \"dafca48f-8187-41c5-b23f-c9b600701701\" (UID: \"dafca48f-8187-41c5-b23f-c9b600701701\") " Oct 30 09:58:51.765180 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:51.765143 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dafca48f-8187-41c5-b23f-c9b600701701-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafca48f-8187-41c5-b23f-c9b600701701-maas-api-openshift-ca\") pod \"dafca48f-8187-41c5-b23f-c9b600701701\" (UID: \"dafca48f-8187-41c5-b23f-c9b600701701\") " Oct 30 09:58:51.765484 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:51.765461 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafca48f-8187-41c5-b23f-c9b600701701-maas-api-openshift-ca" pod "dafca48f-8187-41c5-b23f-c9b600701701" (UID: "dafca48f-8187-41c5-b23f-c9b600701701"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:51.767105 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:51.767085 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafca48f-8187-41c5-b23f-c9b600701701-maas-api-xks-ca" pod "dafca48f-8187-41c5-b23f-c9b600701701" (UID: "dafca48f-8187-41c5-b23f-c9b600701701"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:58:51.767193 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:51.767160 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafca48f-8187-41c5-b23f-c9b600701701-kube-api-access-7pfhv" pod "dafca48f-8187-41c5-b23f-c9b600701701" (UID: "dafca48f-8187-41c5-b23f-c9b600701701"). InnerVolumeSpecName "kube-api-access-7pfhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:51.867037 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:51.867007 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafca48f-8187-41c5-b23f-c9b600701701-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:51.867037 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:51.867031 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pfhv\" (UniqueName: \"kubernetes.io/projected/dafca48f-8187-41c5-b23f-c9b600701701-kube-api-access-7pfhv\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:51.867037 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:51.867041 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafca48f-8187-41c5-b23f-c9b600701701-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:52.502988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:52.502951 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" event={"ID":"dafca48f-8187-41c5-b23f-c9b600701701","Type":"ContainerDied","Data":"d75184b294dc853a8b53ffff865d489ce71d332c0c85c46a12a64cd2a9578696"} Oct 30 09:58:52.502988 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:52.502986 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj" Oct 30 09:58:52.503205 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:52.503009 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d75184b294dc853a8b53ffff865d489ce71d332c0c85c46a12a64cd2a9578696" Oct 30 09:58:53.407938 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:53.407912 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" Oct 30 09:58:53.488602 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:53.488555 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2089f287-3340-45e9-99d4-9e3ce1fbbe9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2089f287-3340-45e9-99d4-9e3ce1fbbe9e-plugins-config-volume\") pod \"2089f287-3340-45e9-99d4-9e3ce1fbbe9e\" (UID: \"2089f287-3340-45e9-99d4-9e3ce1fbbe9e\") " Oct 30 09:58:53.488782 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:53.488621 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2089f287-3340-45e9-99d4-9e3ce1fbbe9e-kube-api-access-49bpd\" (UniqueName: \"kubernetes.io/projected/2089f287-3340-45e9-99d4-9e3ce1fbbe9e-kube-api-access-49bpd\") pod \"2089f287-3340-45e9-99d4-9e3ce1fbbe9e\" (UID: \"2089f287-3340-45e9-99d4-9e3ce1fbbe9e\") " Oct 30 09:58:53.488989 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:53.488962 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2089f287-3340-45e9-99d4-9e3ce1fbbe9e-plugins-config-volume" pod "2089f287-3340-45e9-99d4-9e3ce1fbbe9e" (UID: "2089f287-3340-45e9-99d4-9e3ce1fbbe9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:53.491158 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:53.491131 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2089f287-3340-45e9-99d4-9e3ce1fbbe9e-kube-api-access-49bpd" pod "2089f287-3340-45e9-99d4-9e3ce1fbbe9e" (UID: "2089f287-3340-45e9-99d4-9e3ce1fbbe9e"). InnerVolumeSpecName "kube-api-access-49bpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:53.511929 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:53.511907 5938 generic.go:356] "Generic (PLEG): container finished" podID="2089f287-3340-45e9-99d4-9e3ce1fbbe9e" containerID="4cf35131c08463da4e5264099fd47802b7e80952a9fbcc0621ddc7738a5b6425" exitCode=0 Oct 30 09:58:53.512024 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:53.511978 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" event={"ID":"2089f287-3340-45e9-99d4-9e3ce1fbbe9e","Type":"ContainerDied","Data":"4cf35131c08463da4e5264099fd47802b7e80952a9fbcc0621ddc7738a5b6425"} Oct 30 09:58:53.512024 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:53.512003 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" Oct 30 09:58:53.512024 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:53.512020 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc" event={"ID":"2089f287-3340-45e9-99d4-9e3ce1fbbe9e","Type":"ContainerDied","Data":"89db08a35dd0f7e1dfd2c159be92dd84667751a6a245e30c18e36567b8b78e65"} Oct 30 09:58:53.512131 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:53.512041 5938 scope.go:122] "RemoveContainer" containerID="4cf35131c08463da4e5264099fd47802b7e80952a9fbcc0621ddc7738a5b6425" Oct 30 09:58:53.522933 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:53.522918 5938 scope.go:122] "RemoveContainer" containerID="4cf35131c08463da4e5264099fd47802b7e80952a9fbcc0621ddc7738a5b6425" Oct 30 09:58:53.523222 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:58:53.523204 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf35131c08463da4e5264099fd47802b7e80952a9fbcc0621ddc7738a5b6425\": container with ID starting with 4cf35131c08463da4e5264099fd47802b7e80952a9fbcc0621ddc7738a5b6425 not found: ID does not exist" containerID="4cf35131c08463da4e5264099fd47802b7e80952a9fbcc0621ddc7738a5b6425" Oct 30 09:58:53.523297 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:53.523231 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf35131c08463da4e5264099fd47802b7e80952a9fbcc0621ddc7738a5b6425"} err="failed to get container status \"4cf35131c08463da4e5264099fd47802b7e80952a9fbcc0621ddc7738a5b6425\": rpc error: code = NotFound desc = could not find container \"4cf35131c08463da4e5264099fd47802b7e80952a9fbcc0621ddc7738a5b6425\": container with ID starting with 4cf35131c08463da4e5264099fd47802b7e80952a9fbcc0621ddc7738a5b6425 not found: ID does not exist" Oct 30 09:58:53.569513 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:53.569480 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc"] Oct 30 09:58:53.569707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:53.569525 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f6f96-56498d58d-j2kdc"] Oct 30 09:58:53.589766 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:53.589736 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2089f287-3340-45e9-99d4-9e3ce1fbbe9e-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:53.589766 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:53.589762 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49bpd\" (UniqueName: \"kubernetes.io/projected/2089f287-3340-45e9-99d4-9e3ce1fbbe9e-kube-api-access-49bpd\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:58:54.256699 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:54.256665 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2089f287-3340-45e9-99d4-9e3ce1fbbe9e" path="/var/lib/kubelet/pods/2089f287-3340-45e9-99d4-9e3ce1fbbe9e/volumes" Oct 30 09:58:58.729804 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:58:58.729771 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46"] Oct 30 09:59:01.253048 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:01.253020 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll"] Oct 30 09:59:01.260112 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:01.260082 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-19c94a-wftll"] Oct 30 09:59:02.256403 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:02.256373 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e347845-b234-44ec-9a56-fe279c0913ba" path="/var/lib/kubelet/pods/9e347845-b234-44ec-9a56-fe279c0913ba/volumes" Oct 30 09:59:03.030936 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.030894 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" podUID="52f16458-2419-4f90-a9e4-1d2344e24509" containerName="main" containerID="cri-o://4a945cd4e784d3bba15c7ff434101ff55ac87035fcf1f6dfb6b906bdecb379a3" gracePeriod=45 Oct 30 09:59:03.302149 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.302126 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:59:03.396030 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.396000 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-tmp-dir\") pod \"52f16458-2419-4f90-a9e4-1d2344e24509\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " Oct 30 09:59:03.396209 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.396059 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-home\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-home\") pod \"52f16458-2419-4f90-a9e4-1d2344e24509\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " Oct 30 09:59:03.396209 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.396164 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f16458-2419-4f90-a9e4-1d2344e24509-kube-api-access-jv8np\" (UniqueName: \"kubernetes.io/projected/52f16458-2419-4f90-a9e4-1d2344e24509-kube-api-access-jv8np\") pod \"52f16458-2419-4f90-a9e4-1d2344e24509\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " Oct 30 09:59:03.396339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.396208 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-dshm\") pod \"52f16458-2419-4f90-a9e4-1d2344e24509\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " Oct 30 09:59:03.396339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.396243 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-tmp-dir" pod "52f16458-2419-4f90-a9e4-1d2344e24509" (UID: "52f16458-2419-4f90-a9e4-1d2344e24509"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:03.396339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.396272 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52f16458-2419-4f90-a9e4-1d2344e24509-tls-certs\" (UniqueName: \"kubernetes.io/secret/52f16458-2419-4f90-a9e4-1d2344e24509-tls-certs\") pod \"52f16458-2419-4f90-a9e4-1d2344e24509\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " Oct 30 09:59:03.396339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.396302 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-home" pod "52f16458-2419-4f90-a9e4-1d2344e24509" (UID: "52f16458-2419-4f90-a9e4-1d2344e24509"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:03.396339 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.396319 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-model-cache\") pod \"52f16458-2419-4f90-a9e4-1d2344e24509\" (UID: \"52f16458-2419-4f90-a9e4-1d2344e24509\") " Oct 30 09:59:03.396681 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.396656 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-model-cache" pod "52f16458-2419-4f90-a9e4-1d2344e24509" (UID: "52f16458-2419-4f90-a9e4-1d2344e24509"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:03.396829 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.396811 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:03.397031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.396846 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:03.397031 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.396868 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:03.398577 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.398547 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f16458-2419-4f90-a9e4-1d2344e24509-tls-certs" pod "52f16458-2419-4f90-a9e4-1d2344e24509" (UID: "52f16458-2419-4f90-a9e4-1d2344e24509"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:03.398714 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.398622 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f16458-2419-4f90-a9e4-1d2344e24509-kube-api-access-jv8np" pod "52f16458-2419-4f90-a9e4-1d2344e24509" (UID: "52f16458-2419-4f90-a9e4-1d2344e24509"). InnerVolumeSpecName "kube-api-access-jv8np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:03.398714 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.398654 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-dshm" pod "52f16458-2419-4f90-a9e4-1d2344e24509" (UID: "52f16458-2419-4f90-a9e4-1d2344e24509"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:03.498550 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.498522 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv8np\" (UniqueName: \"kubernetes.io/projected/52f16458-2419-4f90-a9e4-1d2344e24509-kube-api-access-jv8np\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:03.498550 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.498546 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f16458-2419-4f90-a9e4-1d2344e24509-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:03.498730 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.498555 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f16458-2419-4f90-a9e4-1d2344e24509-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:03.586725 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.586671 5938 generic.go:356] "Generic (PLEG): container finished" podID="52f16458-2419-4f90-a9e4-1d2344e24509" containerID="4a945cd4e784d3bba15c7ff434101ff55ac87035fcf1f6dfb6b906bdecb379a3" exitCode=0 Oct 30 09:59:03.586725 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.586719 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" Oct 30 09:59:03.586888 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.586737 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" event={"ID":"52f16458-2419-4f90-a9e4-1d2344e24509","Type":"ContainerDied","Data":"4a945cd4e784d3bba15c7ff434101ff55ac87035fcf1f6dfb6b906bdecb379a3"} Oct 30 09:59:03.586888 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.586766 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt" event={"ID":"52f16458-2419-4f90-a9e4-1d2344e24509","Type":"ContainerDied","Data":"03ecdacc87aa1c8088b8a81a96feed6fbacb21972b60b971622b3ba3169ba816"} Oct 30 09:59:03.586888 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.586783 5938 scope.go:122] "RemoveContainer" containerID="4a945cd4e784d3bba15c7ff434101ff55ac87035fcf1f6dfb6b906bdecb379a3" Oct 30 09:59:03.600668 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.600649 5938 scope.go:122] "RemoveContainer" containerID="4a945cd4e784d3bba15c7ff434101ff55ac87035fcf1f6dfb6b906bdecb379a3" Oct 30 09:59:03.600975 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:59:03.600952 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a945cd4e784d3bba15c7ff434101ff55ac87035fcf1f6dfb6b906bdecb379a3\": container with ID starting with 4a945cd4e784d3bba15c7ff434101ff55ac87035fcf1f6dfb6b906bdecb379a3 not found: ID does not exist" containerID="4a945cd4e784d3bba15c7ff434101ff55ac87035fcf1f6dfb6b906bdecb379a3" Oct 30 09:59:03.601074 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.600980 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a945cd4e784d3bba15c7ff434101ff55ac87035fcf1f6dfb6b906bdecb379a3"} err="failed to get container status \"4a945cd4e784d3bba15c7ff434101ff55ac87035fcf1f6dfb6b906bdecb379a3\": rpc error: code = NotFound desc = could not find container \"4a945cd4e784d3bba15c7ff434101ff55ac87035fcf1f6dfb6b906bdecb379a3\": container with ID starting with 4a945cd4e784d3bba15c7ff434101ff55ac87035fcf1f6dfb6b906bdecb379a3 not found: ID does not exist" Oct 30 09:59:03.641117 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.641098 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt"] Oct 30 09:59:03.644773 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.644752 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a479df/test-model-a479df-kserve-7c98598c69-254jt"] Oct 30 09:59:03.730398 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.730358 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" podUID="9cf4c90b-e480-41a5-9522-045ee3d94955" containerName="payload-processing" containerID="cri-o://d4e69608212cbb147031f247722a5b7891295e37e89616f15e32ea5dd3ca0fe6" gracePeriod=25 Oct 30 09:59:03.962943 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:03.962916 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" Oct 30 09:59:04.105005 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:04.104929 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cf4c90b-e480-41a5-9522-045ee3d94955-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf4c90b-e480-41a5-9522-045ee3d94955-plugins-config-volume\") pod \"9cf4c90b-e480-41a5-9522-045ee3d94955\" (UID: \"9cf4c90b-e480-41a5-9522-045ee3d94955\") " Oct 30 09:59:04.105181 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:04.105043 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf4c90b-e480-41a5-9522-045ee3d94955-kube-api-access-dtszc\" (UniqueName: \"kubernetes.io/projected/9cf4c90b-e480-41a5-9522-045ee3d94955-kube-api-access-dtszc\") pod \"9cf4c90b-e480-41a5-9522-045ee3d94955\" (UID: \"9cf4c90b-e480-41a5-9522-045ee3d94955\") " Oct 30 09:59:04.105310 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:04.105288 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cf4c90b-e480-41a5-9522-045ee3d94955-plugins-config-volume" pod "9cf4c90b-e480-41a5-9522-045ee3d94955" (UID: "9cf4c90b-e480-41a5-9522-045ee3d94955"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:04.107366 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:04.107337 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf4c90b-e480-41a5-9522-045ee3d94955-kube-api-access-dtszc" pod "9cf4c90b-e480-41a5-9522-045ee3d94955" (UID: "9cf4c90b-e480-41a5-9522-045ee3d94955"). InnerVolumeSpecName "kube-api-access-dtszc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:04.206835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:04.206804 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtszc\" (UniqueName: \"kubernetes.io/projected/9cf4c90b-e480-41a5-9522-045ee3d94955-kube-api-access-dtszc\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:04.206835 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:04.206836 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf4c90b-e480-41a5-9522-045ee3d94955-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:04.256297 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:04.256274 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f16458-2419-4f90-a9e4-1d2344e24509" path="/var/lib/kubelet/pods/52f16458-2419-4f90-a9e4-1d2344e24509/volumes" Oct 30 09:59:04.596840 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:04.596810 5938 generic.go:356] "Generic (PLEG): container finished" podID="9cf4c90b-e480-41a5-9522-045ee3d94955" containerID="d4e69608212cbb147031f247722a5b7891295e37e89616f15e32ea5dd3ca0fe6" exitCode=0 Oct 30 09:59:04.597303 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:04.596925 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" event={"ID":"9cf4c90b-e480-41a5-9522-045ee3d94955","Type":"ContainerDied","Data":"d4e69608212cbb147031f247722a5b7891295e37e89616f15e32ea5dd3ca0fe6"} Oct 30 09:59:04.597303 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:04.596952 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" Oct 30 09:59:04.597303 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:04.596987 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46" event={"ID":"9cf4c90b-e480-41a5-9522-045ee3d94955","Type":"ContainerDied","Data":"fefbcf058e499a976a249c68732290d52e7ec5a56956bf4805100f4ea9cd3f33"} Oct 30 09:59:04.597303 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:04.597019 5938 scope.go:122] "RemoveContainer" containerID="d4e69608212cbb147031f247722a5b7891295e37e89616f15e32ea5dd3ca0fe6" Oct 30 09:59:04.607324 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:04.607306 5938 scope.go:122] "RemoveContainer" containerID="d4e69608212cbb147031f247722a5b7891295e37e89616f15e32ea5dd3ca0fe6" Oct 30 09:59:04.607623 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:59:04.607577 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e69608212cbb147031f247722a5b7891295e37e89616f15e32ea5dd3ca0fe6\": container with ID starting with d4e69608212cbb147031f247722a5b7891295e37e89616f15e32ea5dd3ca0fe6 not found: ID does not exist" containerID="d4e69608212cbb147031f247722a5b7891295e37e89616f15e32ea5dd3ca0fe6" Oct 30 09:59:04.607710 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:04.607634 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e69608212cbb147031f247722a5b7891295e37e89616f15e32ea5dd3ca0fe6"} err="failed to get container status \"d4e69608212cbb147031f247722a5b7891295e37e89616f15e32ea5dd3ca0fe6\": rpc error: code = NotFound desc = could not find container \"d4e69608212cbb147031f247722a5b7891295e37e89616f15e32ea5dd3ca0fe6\": container with ID starting with d4e69608212cbb147031f247722a5b7891295e37e89616f15e32ea5dd3ca0fe6 not found: ID does not exist" Oct 30 09:59:04.651026 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:04.651005 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46"] Oct 30 09:59:04.664518 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:04.664490 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82ccfd-85fb77bcf6-k8v46"] Oct 30 09:59:06.256329 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:06.256295 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cf4c90b-e480-41a5-9522-045ee3d94955" path="/var/lib/kubelet/pods/9cf4c90b-e480-41a5-9522-045ee3d94955/volumes" Oct 30 09:59:11.289645 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:11.289612 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj"] Oct 30 09:59:11.302935 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:11.302906 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82ccfd-lb9rj"] Oct 30 09:59:12.256272 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:12.256232 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dafca48f-8187-41c5-b23f-c9b600701701" path="/var/lib/kubelet/pods/dafca48f-8187-41c5-b23f-c9b600701701/volumes" Oct 30 09:59:15.808330 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:59:15.808294 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:59:18.329779 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:18.329742 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz"] Oct 30 09:59:18.330298 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:18.330025 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" podUID="c7325b5f-f90d-44a9-876d-48245c0fe2b0" containerName="istio-proxy" containerID="cri-o://3c14ec9b0900dc0c0ffefa848b0f8f51164cf74a8c85c28d922376f27aebc8f5" gracePeriod=30 Oct 30 09:59:19.636621 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:19.636569 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq"] Oct 30 09:59:19.669169 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:19.669139 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt"] Oct 30 09:59:23.590067 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.590040 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:59:23.710273 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.710244 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-credential-socket\") pod \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " Oct 30 09:59:23.710427 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.710296 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-podinfo\") pod \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " Oct 30 09:59:23.710427 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.710341 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-envoy\") pod \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " Oct 30 09:59:23.710427 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.710394 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-data\") pod \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " Oct 30 09:59:23.710655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.710428 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-workload-certs\") pod \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " Oct 30 09:59:23.710655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.710505 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7325b5f-f90d-44a9-876d-48245c0fe2b0-kube-api-access-2jlc2\" (UniqueName: \"kubernetes.io/projected/c7325b5f-f90d-44a9-876d-48245c0fe2b0-kube-api-access-2jlc2\") pod \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " Oct 30 09:59:23.710655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.710535 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-token\" (UniqueName: \"kubernetes.io/projected/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-token\") pod \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " Oct 30 09:59:23.710655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.710552 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-credential-socket" pod "c7325b5f-f90d-44a9-876d-48245c0fe2b0" (UID: "c7325b5f-f90d-44a9-876d-48245c0fe2b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.710655 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.710561 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istiod-ca-cert\") pod \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " Oct 30 09:59:23.710964 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.710682 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-workload-socket\") pod \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\" (UID: \"c7325b5f-f90d-44a9-876d-48245c0fe2b0\") " Oct 30 09:59:23.711030 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.710980 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-workload-certs" pod "c7325b5f-f90d-44a9-876d-48245c0fe2b0" (UID: "c7325b5f-f90d-44a9-876d-48245c0fe2b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.711030 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.710992 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-workload-socket" pod "c7325b5f-f90d-44a9-876d-48245c0fe2b0" (UID: "c7325b5f-f90d-44a9-876d-48245c0fe2b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.711228 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.711202 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-data" pod "c7325b5f-f90d-44a9-876d-48245c0fe2b0" (UID: "c7325b5f-f90d-44a9-876d-48245c0fe2b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.711353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.711268 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.711353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.711292 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.711353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.711308 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.711638 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.711615 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istiod-ca-cert" pod "c7325b5f-f90d-44a9-876d-48245c0fe2b0" (UID: "c7325b5f-f90d-44a9-876d-48245c0fe2b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:23.713115 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.713090 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-podinfo" pod "c7325b5f-f90d-44a9-876d-48245c0fe2b0" (UID: "c7325b5f-f90d-44a9-876d-48245c0fe2b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:59:23.713254 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.713235 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-envoy" pod "c7325b5f-f90d-44a9-876d-48245c0fe2b0" (UID: "c7325b5f-f90d-44a9-876d-48245c0fe2b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.713816 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.713786 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-token" pod "c7325b5f-f90d-44a9-876d-48245c0fe2b0" (UID: "c7325b5f-f90d-44a9-876d-48245c0fe2b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:23.713924 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.713824 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7325b5f-f90d-44a9-876d-48245c0fe2b0-kube-api-access-2jlc2" pod "c7325b5f-f90d-44a9-876d-48245c0fe2b0" (UID: "c7325b5f-f90d-44a9-876d-48245c0fe2b0"). InnerVolumeSpecName "kube-api-access-2jlc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:23.745495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.745468 5938 generic.go:356] "Generic (PLEG): container finished" podID="c7325b5f-f90d-44a9-876d-48245c0fe2b0" containerID="3c14ec9b0900dc0c0ffefa848b0f8f51164cf74a8c85c28d922376f27aebc8f5" exitCode=0 Oct 30 09:59:23.745646 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.745529 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" Oct 30 09:59:23.745646 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.745563 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" event={"ID":"c7325b5f-f90d-44a9-876d-48245c0fe2b0","Type":"ContainerDied","Data":"3c14ec9b0900dc0c0ffefa848b0f8f51164cf74a8c85c28d922376f27aebc8f5"} Oct 30 09:59:23.745780 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.745646 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz" event={"ID":"c7325b5f-f90d-44a9-876d-48245c0fe2b0","Type":"ContainerDied","Data":"d6576915d06f3b4b35e5c49c2c89133b4c5845a553cdbc99c1bddeb89be42ba4"} Oct 30 09:59:23.745780 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.745680 5938 scope.go:122] "RemoveContainer" containerID="3c14ec9b0900dc0c0ffefa848b0f8f51164cf74a8c85c28d922376f27aebc8f5" Oct 30 09:59:23.760539 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.760083 5938 scope.go:122] "RemoveContainer" containerID="3c14ec9b0900dc0c0ffefa848b0f8f51164cf74a8c85c28d922376f27aebc8f5" Oct 30 09:59:23.760539 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:59:23.760465 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c14ec9b0900dc0c0ffefa848b0f8f51164cf74a8c85c28d922376f27aebc8f5\": container with ID starting with 3c14ec9b0900dc0c0ffefa848b0f8f51164cf74a8c85c28d922376f27aebc8f5 not found: ID does not exist" containerID="3c14ec9b0900dc0c0ffefa848b0f8f51164cf74a8c85c28d922376f27aebc8f5" Oct 30 09:59:23.760539 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.760491 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c14ec9b0900dc0c0ffefa848b0f8f51164cf74a8c85c28d922376f27aebc8f5"} err="failed to get container status \"3c14ec9b0900dc0c0ffefa848b0f8f51164cf74a8c85c28d922376f27aebc8f5\": rpc error: code = NotFound desc = could not find container \"3c14ec9b0900dc0c0ffefa848b0f8f51164cf74a8c85c28d922376f27aebc8f5\": container with ID starting with 3c14ec9b0900dc0c0ffefa848b0f8f51164cf74a8c85c28d922376f27aebc8f5 not found: ID does not exist" Oct 30 09:59:23.801169 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.801121 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz"] Oct 30 09:59:23.803755 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.803729 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-xhsvz"] Oct 30 09:59:23.813056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.813034 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jlc2\" (UniqueName: \"kubernetes.io/projected/c7325b5f-f90d-44a9-876d-48245c0fe2b0-kube-api-access-2jlc2\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.813056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.813055 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.813190 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.813065 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.813190 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.813073 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.813190 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.813080 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.813190 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:23.813088 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7325b5f-f90d-44a9-876d-48245c0fe2b0-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:24.256056 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:24.256024 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7325b5f-f90d-44a9-876d-48245c0fe2b0" path="/var/lib/kubelet/pods/c7325b5f-f90d-44a9-876d-48245c0fe2b0/volumes" Oct 30 09:59:24.637497 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:24.637453 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" podUID="2d4172ae-9665-4707-97ab-6b3c82fb7242" containerName="payload-processing" containerID="cri-o://d6f7eb481f337708f2d0f6d8eb6bb58d2de4de1d57b626e2aa48452d65ea9e51" gracePeriod=25 Oct 30 09:59:24.669525 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:24.669488 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" podUID="5ba161e1-40bc-4ea2-858a-44b63d9bdc83" containerName="payload-pre-processing" containerID="cri-o://123d2e119810a1b0cf78e17c3e550b596c80f2d406cca575a62d34206210910d" gracePeriod=25 Oct 30 09:59:29.419224 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:29.419191 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s"] Oct 30 09:59:29.454734 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:29.454707 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l"] Oct 30 09:59:33.667604 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:33.667560 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" podUID="2d4172ae-9665-4707-97ab-6b3c82fb7242" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:59:33.668028 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:33.667707 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" podUID="5ba161e1-40bc-4ea2-858a-44b63d9bdc83" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:59:33.795105 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:33.795078 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf"] Oct 30 09:59:33.809524 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:33.809486 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv"] Oct 30 09:59:33.811961 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:33.811938 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq"] Oct 30 09:59:34.420381 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:34.420310 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" podUID="80bc4ca3-dd13-4fa4-900f-82b93b520ffe" containerName="payload-processing" containerID="cri-o://8fba50c95fdca4091671a1c052d2a56fe007f6f558e474711dd72994e71272e7" gracePeriod=25 Oct 30 09:59:34.455910 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:34.455843 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" podUID="e9c527fc-edd4-4a04-a781-483921b54c6b" containerName="payload-pre-processing" containerID="cri-o://99845b450fe48ef8f15f68f6928ff521334a9778ee9179df21243739d1e2490f" gracePeriod=25 Oct 30 09:59:42.647761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:42.647709 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" podUID="e9c527fc-edd4-4a04-a781-483921b54c6b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:59:43.655311 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:43.655273 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" podUID="80bc4ca3-dd13-4fa4-900f-82b93b520ffe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:59:43.667331 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:43.667294 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" podUID="5ba161e1-40bc-4ea2-858a-44b63d9bdc83" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:59:43.667507 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:43.667484 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" podUID="2d4172ae-9665-4707-97ab-6b3c82fb7242" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:59:44.750269 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:44.750244 5938 scope.go:122] "RemoveContainer" containerID="4a59bdd55838946351d7f21d20c13a928c1f6b3f91ef18750e4a5812de49365c" Oct 30 09:59:44.784066 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:44.784042 5938 scope.go:122] "RemoveContainer" containerID="3ffd0b2714207ba494a6cca997722003c0aba53270dbad515f6aab8c2a502128" Oct 30 09:59:44.795183 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:44.795148 5938 scope.go:122] "RemoveContainer" containerID="a4a4413ca29138c58f17837ca209c9a7cef9692db604445d8384a7d534d4f25d" Oct 30 09:59:44.815345 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:44.815317 5938 scope.go:122] "RemoveContainer" containerID="032374664dd0f4f9078b2c6dc9cd2adae1f58165b630a4b3b9e424f514376960" Oct 30 09:59:44.826316 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:44.826297 5938 scope.go:122] "RemoveContainer" containerID="ce35963cc91e32b2c69b49e87f48a92717c2f983753a629309a870080d3b0ba5" Oct 30 09:59:44.841682 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:44.840952 5938 scope.go:122] "RemoveContainer" containerID="f1911f620a59761db1f89d10085c520dad2813c9af54ddd9dc1818579c4fc4b6" Oct 30 09:59:44.863761 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:44.863725 5938 scope.go:122] "RemoveContainer" containerID="57aacc5a6e3b004cdcaff613ea471059d2e699282b881264b496937e15c7d380" Oct 30 09:59:44.874469 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:44.874445 5938 scope.go:122] "RemoveContainer" containerID="e8808b2301dde0ad62942a53e060d911d2890c7f114f85c9f59e48dbd91cea25" Oct 30 09:59:44.890392 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:44.890369 5938 scope.go:122] "RemoveContainer" containerID="916970aa2dc1d8cdff84056a524466344273f94bb89ec33da8adb032073c977d" Oct 30 09:59:44.905114 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:44.905030 5938 scope.go:122] "RemoveContainer" containerID="80392f2d2b3b9eb511a6918de3cab20f06e4eee48fae40e707963f3c361b5dd2" Oct 30 09:59:44.909309 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:44.909288 5938 generic.go:356] "Generic (PLEG): container finished" podID="5ba161e1-40bc-4ea2-858a-44b63d9bdc83" containerID="123d2e119810a1b0cf78e17c3e550b596c80f2d406cca575a62d34206210910d" exitCode=0 Oct 30 09:59:44.909441 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:44.909419 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" event={"ID":"5ba161e1-40bc-4ea2-858a-44b63d9bdc83","Type":"ContainerDied","Data":"123d2e119810a1b0cf78e17c3e550b596c80f2d406cca575a62d34206210910d"} Oct 30 09:59:44.911219 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:44.911197 5938 generic.go:356] "Generic (PLEG): container finished" podID="2d4172ae-9665-4707-97ab-6b3c82fb7242" containerID="d6f7eb481f337708f2d0f6d8eb6bb58d2de4de1d57b626e2aa48452d65ea9e51" exitCode=0 Oct 30 09:59:44.911379 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:44.911222 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" event={"ID":"2d4172ae-9665-4707-97ab-6b3c82fb7242","Type":"ContainerDied","Data":"d6f7eb481f337708f2d0f6d8eb6bb58d2de4de1d57b626e2aa48452d65ea9e51"} Oct 30 09:59:44.951791 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:44.951767 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" Oct 30 09:59:44.984963 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:44.984946 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" Oct 30 09:59:45.033189 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.033166 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d4172ae-9665-4707-97ab-6b3c82fb7242-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4172ae-9665-4707-97ab-6b3c82fb7242-plugins-config-volume\") pod \"2d4172ae-9665-4707-97ab-6b3c82fb7242\" (UID: \"2d4172ae-9665-4707-97ab-6b3c82fb7242\") " Oct 30 09:59:45.033318 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.033289 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4172ae-9665-4707-97ab-6b3c82fb7242-kube-api-access-5h2gj\" (UniqueName: \"kubernetes.io/projected/2d4172ae-9665-4707-97ab-6b3c82fb7242-kube-api-access-5h2gj\") pod \"2d4172ae-9665-4707-97ab-6b3c82fb7242\" (UID: \"2d4172ae-9665-4707-97ab-6b3c82fb7242\") " Oct 30 09:59:45.033580 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.033558 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4172ae-9665-4707-97ab-6b3c82fb7242-plugins-config-volume" pod "2d4172ae-9665-4707-97ab-6b3c82fb7242" (UID: "2d4172ae-9665-4707-97ab-6b3c82fb7242"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:45.033770 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.033747 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4172ae-9665-4707-97ab-6b3c82fb7242-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:45.035653 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.035629 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4172ae-9665-4707-97ab-6b3c82fb7242-kube-api-access-5h2gj" pod "2d4172ae-9665-4707-97ab-6b3c82fb7242" (UID: "2d4172ae-9665-4707-97ab-6b3c82fb7242"). InnerVolumeSpecName "kube-api-access-5h2gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:45.135116 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.135075 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba161e1-40bc-4ea2-858a-44b63d9bdc83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba161e1-40bc-4ea2-858a-44b63d9bdc83-plugins-config-volume\") pod \"5ba161e1-40bc-4ea2-858a-44b63d9bdc83\" (UID: \"5ba161e1-40bc-4ea2-858a-44b63d9bdc83\") " Oct 30 09:59:45.135308 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.135148 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba161e1-40bc-4ea2-858a-44b63d9bdc83-kube-api-access-b8gxp\" (UniqueName: \"kubernetes.io/projected/5ba161e1-40bc-4ea2-858a-44b63d9bdc83-kube-api-access-b8gxp\") pod \"5ba161e1-40bc-4ea2-858a-44b63d9bdc83\" (UID: \"5ba161e1-40bc-4ea2-858a-44b63d9bdc83\") " Oct 30 09:59:45.135448 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.135433 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h2gj\" (UniqueName: \"kubernetes.io/projected/2d4172ae-9665-4707-97ab-6b3c82fb7242-kube-api-access-5h2gj\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:45.135581 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.135557 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba161e1-40bc-4ea2-858a-44b63d9bdc83-plugins-config-volume" pod "5ba161e1-40bc-4ea2-858a-44b63d9bdc83" (UID: "5ba161e1-40bc-4ea2-858a-44b63d9bdc83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:45.137736 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.137710 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba161e1-40bc-4ea2-858a-44b63d9bdc83-kube-api-access-b8gxp" pod "5ba161e1-40bc-4ea2-858a-44b63d9bdc83" (UID: "5ba161e1-40bc-4ea2-858a-44b63d9bdc83"). InnerVolumeSpecName "kube-api-access-b8gxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:45.236385 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.236319 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba161e1-40bc-4ea2-858a-44b63d9bdc83-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:45.236385 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.236369 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8gxp\" (UniqueName: \"kubernetes.io/projected/5ba161e1-40bc-4ea2-858a-44b63d9bdc83-kube-api-access-b8gxp\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:45.810550 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:59:45.810513 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 09:59:45.925106 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.925032 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" Oct 30 09:59:45.925106 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.925061 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt" event={"ID":"5ba161e1-40bc-4ea2-858a-44b63d9bdc83","Type":"ContainerDied","Data":"65ee762ea0778b9f8bc5e69de542b0a12089dcdcc593e4f0953b06b99a245430"} Oct 30 09:59:45.925321 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.925129 5938 scope.go:122] "RemoveContainer" containerID="123d2e119810a1b0cf78e17c3e550b596c80f2d406cca575a62d34206210910d" Oct 30 09:59:45.926830 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.926814 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" Oct 30 09:59:45.926975 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.926910 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq" event={"ID":"2d4172ae-9665-4707-97ab-6b3c82fb7242","Type":"ContainerDied","Data":"5a8770cd5122dff29c6cfeea061e99fd83f3b7e37ba464b9dc6ad3693c04bfe6"} Oct 30 09:59:45.936342 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.936318 5938 scope.go:122] "RemoveContainer" containerID="d6f7eb481f337708f2d0f6d8eb6bb58d2de4de1d57b626e2aa48452d65ea9e51" Oct 30 09:59:45.979191 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.979166 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt"] Oct 30 09:59:45.993377 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:45.993353 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c88ae-55cccd4c87-s24xt"] Oct 30 09:59:46.030499 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:46.030429 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq"] Oct 30 09:59:46.032742 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:46.032721 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7c88ae-b5798c666-c22gq"] Oct 30 09:59:46.256150 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:46.256066 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d4172ae-9665-4707-97ab-6b3c82fb7242" path="/var/lib/kubelet/pods/2d4172ae-9665-4707-97ab-6b3c82fb7242/volumes" Oct 30 09:59:46.256534 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:46.256512 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba161e1-40bc-4ea2-858a-44b63d9bdc83" path="/var/lib/kubelet/pods/5ba161e1-40bc-4ea2-858a-44b63d9bdc83/volumes" Oct 30 09:59:48.819707 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:48.819662 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" podUID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" containerName="main" containerID="cri-o://6e43cf073bfcb9320704b0ace76d6bb48bdba976d0420bc5d0604b1a87c39267" gracePeriod=45 Oct 30 09:59:48.832957 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:48.832926 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" containerID="cri-o://c5ed09cfe8fcd7902d2ebc569605df0885586deea90832a5b4584d3e55a607ac" gracePeriod=45 Oct 30 09:59:48.834929 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:48.834897 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" containerID="cri-o://223df8d825a01a7d078596148b24ddced81bdaad40c3f2cd9733a2179a3abe93" gracePeriod=45 Oct 30 09:59:48.950888 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:48.950856 5938 generic.go:356] "Generic (PLEG): container finished" podID="372a7ce3-4393-48e0-a192-10e0530e933d" containerID="c5ed09cfe8fcd7902d2ebc569605df0885586deea90832a5b4584d3e55a607ac" exitCode=0 Oct 30 09:59:48.951021 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:48.950924 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" event={"ID":"372a7ce3-4393-48e0-a192-10e0530e933d","Type":"ContainerDied","Data":"c5ed09cfe8fcd7902d2ebc569605df0885586deea90832a5b4584d3e55a607ac"} Oct 30 09:59:48.953108 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:48.953081 5938 generic.go:356] "Generic (PLEG): container finished" podID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" containerID="6e43cf073bfcb9320704b0ace76d6bb48bdba976d0420bc5d0604b1a87c39267" exitCode=0 Oct 30 09:59:48.953245 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:48.953176 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" event={"ID":"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7","Type":"ContainerDied","Data":"6e43cf073bfcb9320704b0ace76d6bb48bdba976d0420bc5d0604b1a87c39267"} Oct 30 09:59:49.146495 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.146470 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:59:49.149859 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.149838 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:59:49.153715 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.153694 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:59:49.279353 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.279320 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2371dfe4-c1c9-4773-b650-4d28e86a2f08-tls-certs\" (UniqueName: \"kubernetes.io/secret/2371dfe4-c1c9-4773-b650-4d28e86a2f08-tls-certs\") pod \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " Oct 30 09:59:49.279531 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.279363 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/372a7ce3-4393-48e0-a192-10e0530e933d-tls-certs\" (UniqueName: \"kubernetes.io/secret/372a7ce3-4393-48e0-a192-10e0530e933d-tls-certs\") pod \"372a7ce3-4393-48e0-a192-10e0530e933d\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " Oct 30 09:59:49.279531 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.279402 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-model-cache\") pod \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " Oct 30 09:59:49.279531 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.279445 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-tls-certs\" (UniqueName: \"kubernetes.io/secret/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-tls-certs\") pod \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " Oct 30 09:59:49.279531 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.279473 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-home\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-home\") pod \"372a7ce3-4393-48e0-a192-10e0530e933d\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " Oct 30 09:59:49.279531 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.279509 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-model-cache\") pod \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " Oct 30 09:59:49.279971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.279541 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-model-cache\") pod \"372a7ce3-4393-48e0-a192-10e0530e933d\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " Oct 30 09:59:49.279971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.279619 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/372a7ce3-4393-48e0-a192-10e0530e933d-kube-api-access-vvh9x\" (UniqueName: \"kubernetes.io/projected/372a7ce3-4393-48e0-a192-10e0530e933d-kube-api-access-vvh9x\") pod \"372a7ce3-4393-48e0-a192-10e0530e933d\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " Oct 30 09:59:49.279971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.279660 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-dshm\") pod \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " Oct 30 09:59:49.279971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.279686 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-tmp-dir\") pod \"372a7ce3-4393-48e0-a192-10e0530e933d\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " Oct 30 09:59:49.279971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.279714 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-dshm\") pod \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " Oct 30 09:59:49.279971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.279760 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-tmp-dir\") pod \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " Oct 30 09:59:49.279971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.279867 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-tmp-dir\") pod \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " Oct 30 09:59:49.279971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.279911 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-dshm\") pod \"372a7ce3-4393-48e0-a192-10e0530e933d\" (UID: \"372a7ce3-4393-48e0-a192-10e0530e933d\") " Oct 30 09:59:49.279971 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.279945 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-home\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-home\") pod \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " Oct 30 09:59:49.280432 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.279981 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-kube-api-access-2tm8t\" (UniqueName: \"kubernetes.io/projected/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-kube-api-access-2tm8t\") pod \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " Oct 30 09:59:49.280432 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.280029 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2371dfe4-c1c9-4773-b650-4d28e86a2f08-kube-api-access-ldkxf\" (UniqueName: \"kubernetes.io/projected/2371dfe4-c1c9-4773-b650-4d28e86a2f08-kube-api-access-ldkxf\") pod \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\" (UID: \"2371dfe4-c1c9-4773-b650-4d28e86a2f08\") " Oct 30 09:59:49.280432 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.280060 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-home\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-home\") pod \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\" (UID: \"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7\") " Oct 30 09:59:49.281004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.280911 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-tmp-dir" pod "99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" (UID: "99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:49.281004 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.280909 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-tmp-dir" pod "2371dfe4-c1c9-4773-b650-4d28e86a2f08" (UID: "2371dfe4-c1c9-4773-b650-4d28e86a2f08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:49.281370 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.281343 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-tmp-dir" pod "372a7ce3-4393-48e0-a192-10e0530e933d" (UID: "372a7ce3-4393-48e0-a192-10e0530e933d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:49.281918 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.281890 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-home" pod "2371dfe4-c1c9-4773-b650-4d28e86a2f08" (UID: "2371dfe4-c1c9-4773-b650-4d28e86a2f08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:49.282529 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.282369 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-home" pod "99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" (UID: "99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:49.282733 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.282699 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-model-cache" pod "372a7ce3-4393-48e0-a192-10e0530e933d" (UID: "372a7ce3-4393-48e0-a192-10e0530e933d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:49.282814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.282730 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-home" pod "372a7ce3-4393-48e0-a192-10e0530e933d" (UID: "372a7ce3-4393-48e0-a192-10e0530e933d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:49.282814 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.282790 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-model-cache" pod "2371dfe4-c1c9-4773-b650-4d28e86a2f08" (UID: "2371dfe4-c1c9-4773-b650-4d28e86a2f08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:49.283044 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.283008 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-model-cache" pod "99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" (UID: "99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:49.284054 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.284024 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2371dfe4-c1c9-4773-b650-4d28e86a2f08-kube-api-access-ldkxf" pod "2371dfe4-c1c9-4773-b650-4d28e86a2f08" (UID: "2371dfe4-c1c9-4773-b650-4d28e86a2f08"). InnerVolumeSpecName "kube-api-access-ldkxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:49.284298 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.284262 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-kube-api-access-2tm8t" pod "99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" (UID: "99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7"). InnerVolumeSpecName "kube-api-access-2tm8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:49.284431 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.284405 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-dshm" pod "2371dfe4-c1c9-4773-b650-4d28e86a2f08" (UID: "2371dfe4-c1c9-4773-b650-4d28e86a2f08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:49.284896 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.284872 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2371dfe4-c1c9-4773-b650-4d28e86a2f08-tls-certs" pod "2371dfe4-c1c9-4773-b650-4d28e86a2f08" (UID: "2371dfe4-c1c9-4773-b650-4d28e86a2f08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:49.285365 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.285344 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372a7ce3-4393-48e0-a192-10e0530e933d-kube-api-access-vvh9x" pod "372a7ce3-4393-48e0-a192-10e0530e933d" (UID: "372a7ce3-4393-48e0-a192-10e0530e933d"). InnerVolumeSpecName "kube-api-access-vvh9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:49.285519 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.285502 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-dshm" pod "99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" (UID: "99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:49.286491 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.286473 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-tls-certs" pod "99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" (UID: "99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:49.287169 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.287150 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-dshm" pod "372a7ce3-4393-48e0-a192-10e0530e933d" (UID: "372a7ce3-4393-48e0-a192-10e0530e933d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:49.287387 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.287369 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/372a7ce3-4393-48e0-a192-10e0530e933d-tls-certs" pod "372a7ce3-4393-48e0-a192-10e0530e933d" (UID: "372a7ce3-4393-48e0-a192-10e0530e933d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:49.381941 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.381912 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.381941 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.381949 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.382157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.381964 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.382157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.381976 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tm8t\" (UniqueName: \"kubernetes.io/projected/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-kube-api-access-2tm8t\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.382157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.381990 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldkxf\" (UniqueName: \"kubernetes.io/projected/2371dfe4-c1c9-4773-b650-4d28e86a2f08-kube-api-access-ldkxf\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.382157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.382003 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.382157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.382015 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2371dfe4-c1c9-4773-b650-4d28e86a2f08-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.382157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.382027 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372a7ce3-4393-48e0-a192-10e0530e933d-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.382157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.382039 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.382157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.382052 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.382157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.382064 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.382157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.382080 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.382157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.382088 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.382157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.382095 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvh9x\" (UniqueName: \"kubernetes.io/projected/372a7ce3-4393-48e0-a192-10e0530e933d-kube-api-access-vvh9x\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.382157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.382102 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2371dfe4-c1c9-4773-b650-4d28e86a2f08-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.382157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.382110 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372a7ce3-4393-48e0-a192-10e0530e933d-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.382157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.382117 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.382157 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.382125 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.961693 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.961657 5938 generic.go:356] "Generic (PLEG): container finished" podID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerID="223df8d825a01a7d078596148b24ddced81bdaad40c3f2cd9733a2179a3abe93" exitCode=0 Oct 30 09:59:49.962147 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.961705 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" Oct 30 09:59:49.962147 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.961740 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" event={"ID":"2371dfe4-c1c9-4773-b650-4d28e86a2f08","Type":"ContainerDied","Data":"223df8d825a01a7d078596148b24ddced81bdaad40c3f2cd9733a2179a3abe93"} Oct 30 09:59:49.962147 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.961784 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq" event={"ID":"2371dfe4-c1c9-4773-b650-4d28e86a2f08","Type":"ContainerDied","Data":"8b6b6e9fa64a79992932dd2d4e8fb12988ea773ea65177ccc5f4a4a5b570448a"} Oct 30 09:59:49.962147 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.961806 5938 scope.go:122] "RemoveContainer" containerID="223df8d825a01a7d078596148b24ddced81bdaad40c3f2cd9733a2179a3abe93" Oct 30 09:59:49.963712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.963690 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" Oct 30 09:59:49.963712 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.963697 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv" event={"ID":"372a7ce3-4393-48e0-a192-10e0530e933d","Type":"ContainerDied","Data":"270c69443b10343c19aac0beb25cc5cc557448c295b6c9514f45d100903f1411"} Oct 30 09:59:49.965333 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.965308 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" event={"ID":"99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7","Type":"ContainerDied","Data":"8bbffc02bfd168dd022d8d1ec69da36b6e3db3225e8a25418d0a1a6c81984db2"} Oct 30 09:59:49.965467 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.965367 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" Oct 30 09:59:49.974581 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.974563 5938 scope.go:122] "RemoveContainer" containerID="223df8d825a01a7d078596148b24ddced81bdaad40c3f2cd9733a2179a3abe93" Oct 30 09:59:49.974921 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:59:49.974899 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"223df8d825a01a7d078596148b24ddced81bdaad40c3f2cd9733a2179a3abe93\": container with ID starting with 223df8d825a01a7d078596148b24ddced81bdaad40c3f2cd9733a2179a3abe93 not found: ID does not exist" containerID="223df8d825a01a7d078596148b24ddced81bdaad40c3f2cd9733a2179a3abe93" Oct 30 09:59:49.975033 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.974927 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223df8d825a01a7d078596148b24ddced81bdaad40c3f2cd9733a2179a3abe93"} err="failed to get container status \"223df8d825a01a7d078596148b24ddced81bdaad40c3f2cd9733a2179a3abe93\": rpc error: code = NotFound desc = could not find container \"223df8d825a01a7d078596148b24ddced81bdaad40c3f2cd9733a2179a3abe93\": container with ID starting with 223df8d825a01a7d078596148b24ddced81bdaad40c3f2cd9733a2179a3abe93 not found: ID does not exist" Oct 30 09:59:49.975033 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.974944 5938 scope.go:122] "RemoveContainer" containerID="c5ed09cfe8fcd7902d2ebc569605df0885586deea90832a5b4584d3e55a607ac" Oct 30 09:59:49.984713 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:49.984696 5938 scope.go:122] "RemoveContainer" containerID="6e43cf073bfcb9320704b0ace76d6bb48bdba976d0420bc5d0604b1a87c39267" Oct 30 09:59:50.015929 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:50.015908 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv"] Oct 30 09:59:50.029190 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:50.029166 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/premium-w4-3a61e1-kserve-7b667c468c-6gdtv"] Oct 30 09:59:50.066288 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:50.066259 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq"] Oct 30 09:59:50.070788 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:50.070769 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/simulator-w4-3a61e1-kserve-74cfcdb694-vx5qq"] Oct 30 09:59:50.071940 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:50.071920 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf" podUID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:59:50.107362 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:50.107286 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf"] Oct 30 09:59:50.110346 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:50.110317 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/distinct-w4-3a61e1-kserve-76d98bdb78-jdkhf"] Oct 30 09:59:50.256544 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:50.256509 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" path="/var/lib/kubelet/pods/2371dfe4-c1c9-4773-b650-4d28e86a2f08/volumes" Oct 30 09:59:50.257060 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:50.257043 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" path="/var/lib/kubelet/pods/372a7ce3-4393-48e0-a192-10e0530e933d/volumes" Oct 30 09:59:50.257442 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:50.257430 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" path="/var/lib/kubelet/pods/99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7/volumes" Oct 30 09:59:52.648332 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:52.648291 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" podUID="e9c527fc-edd4-4a04-a781-483921b54c6b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:59:53.656000 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:53.655962 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" podUID="80bc4ca3-dd13-4fa4-900f-82b93b520ffe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:59:54.722674 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:54.722649 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" Oct 30 09:59:54.738351 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:54.738330 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80bc4ca3-dd13-4fa4-900f-82b93b520ffe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80bc4ca3-dd13-4fa4-900f-82b93b520ffe-plugins-config-volume\") pod \"80bc4ca3-dd13-4fa4-900f-82b93b520ffe\" (UID: \"80bc4ca3-dd13-4fa4-900f-82b93b520ffe\") " Oct 30 09:59:54.738482 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:54.738370 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80bc4ca3-dd13-4fa4-900f-82b93b520ffe-kube-api-access-qtlm4\" (UniqueName: \"kubernetes.io/projected/80bc4ca3-dd13-4fa4-900f-82b93b520ffe-kube-api-access-qtlm4\") pod \"80bc4ca3-dd13-4fa4-900f-82b93b520ffe\" (UID: \"80bc4ca3-dd13-4fa4-900f-82b93b520ffe\") " Oct 30 09:59:54.738793 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:54.738772 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80bc4ca3-dd13-4fa4-900f-82b93b520ffe-plugins-config-volume" pod "80bc4ca3-dd13-4fa4-900f-82b93b520ffe" (UID: "80bc4ca3-dd13-4fa4-900f-82b93b520ffe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:54.738902 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:54.738887 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80bc4ca3-dd13-4fa4-900f-82b93b520ffe-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:54.739415 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:54.739393 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" Oct 30 09:59:54.740642 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:54.740621 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80bc4ca3-dd13-4fa4-900f-82b93b520ffe-kube-api-access-qtlm4" pod "80bc4ca3-dd13-4fa4-900f-82b93b520ffe" (UID: "80bc4ca3-dd13-4fa4-900f-82b93b520ffe"). InnerVolumeSpecName "kube-api-access-qtlm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:54.839727 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:54.839698 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c527fc-edd4-4a04-a781-483921b54c6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c527fc-edd4-4a04-a781-483921b54c6b-plugins-config-volume\") pod \"e9c527fc-edd4-4a04-a781-483921b54c6b\" (UID: \"e9c527fc-edd4-4a04-a781-483921b54c6b\") " Oct 30 09:59:54.839891 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:54.839759 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c527fc-edd4-4a04-a781-483921b54c6b-kube-api-access-7df54\" (UniqueName: \"kubernetes.io/projected/e9c527fc-edd4-4a04-a781-483921b54c6b-kube-api-access-7df54\") pod \"e9c527fc-edd4-4a04-a781-483921b54c6b\" (UID: \"e9c527fc-edd4-4a04-a781-483921b54c6b\") " Oct 30 09:59:54.840066 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:54.840052 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtlm4\" (UniqueName: \"kubernetes.io/projected/80bc4ca3-dd13-4fa4-900f-82b93b520ffe-kube-api-access-qtlm4\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:54.840172 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:54.840148 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c527fc-edd4-4a04-a781-483921b54c6b-plugins-config-volume" pod "e9c527fc-edd4-4a04-a781-483921b54c6b" (UID: "e9c527fc-edd4-4a04-a781-483921b54c6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:54.842020 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:54.842002 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c527fc-edd4-4a04-a781-483921b54c6b-kube-api-access-7df54" pod "e9c527fc-edd4-4a04-a781-483921b54c6b" (UID: "e9c527fc-edd4-4a04-a781-483921b54c6b"). InnerVolumeSpecName "kube-api-access-7df54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:54.941067 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:54.941040 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c527fc-edd4-4a04-a781-483921b54c6b-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:54.941067 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:54.941064 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7df54\" (UniqueName: \"kubernetes.io/projected/e9c527fc-edd4-4a04-a781-483921b54c6b-kube-api-access-7df54\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 09:59:55.005136 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.005065 5938 generic.go:356] "Generic (PLEG): container finished" podID="e9c527fc-edd4-4a04-a781-483921b54c6b" containerID="99845b450fe48ef8f15f68f6928ff521334a9778ee9179df21243739d1e2490f" exitCode=0 Oct 30 09:59:55.005136 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.005106 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" Oct 30 09:59:55.005333 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.005151 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" event={"ID":"e9c527fc-edd4-4a04-a781-483921b54c6b","Type":"ContainerDied","Data":"99845b450fe48ef8f15f68f6928ff521334a9778ee9179df21243739d1e2490f"} Oct 30 09:59:55.005333 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.005198 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l" event={"ID":"e9c527fc-edd4-4a04-a781-483921b54c6b","Type":"ContainerDied","Data":"d52085f52da729755d7ac869d91d1c36c26231813b8773d39f73222ef15bed45"} Oct 30 09:59:55.005333 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.005220 5938 scope.go:122] "RemoveContainer" containerID="99845b450fe48ef8f15f68f6928ff521334a9778ee9179df21243739d1e2490f" Oct 30 09:59:55.007006 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.006985 5938 generic.go:356] "Generic (PLEG): container finished" podID="80bc4ca3-dd13-4fa4-900f-82b93b520ffe" containerID="8fba50c95fdca4091671a1c052d2a56fe007f6f558e474711dd72994e71272e7" exitCode=0 Oct 30 09:59:55.007136 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.007027 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" Oct 30 09:59:55.007136 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.007077 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" event={"ID":"80bc4ca3-dd13-4fa4-900f-82b93b520ffe","Type":"ContainerDied","Data":"8fba50c95fdca4091671a1c052d2a56fe007f6f558e474711dd72994e71272e7"} Oct 30 09:59:55.007136 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.007113 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s" event={"ID":"80bc4ca3-dd13-4fa4-900f-82b93b520ffe","Type":"ContainerDied","Data":"c1db1804633c6f38bdfa8bc3c417a7a72c768225da6c264728327818556c62a1"} Oct 30 09:59:55.016757 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.016743 5938 scope.go:122] "RemoveContainer" containerID="99845b450fe48ef8f15f68f6928ff521334a9778ee9179df21243739d1e2490f" Oct 30 09:59:55.017071 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:59:55.017050 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99845b450fe48ef8f15f68f6928ff521334a9778ee9179df21243739d1e2490f\": container with ID starting with 99845b450fe48ef8f15f68f6928ff521334a9778ee9179df21243739d1e2490f not found: ID does not exist" containerID="99845b450fe48ef8f15f68f6928ff521334a9778ee9179df21243739d1e2490f" Oct 30 09:59:55.017185 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.017087 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99845b450fe48ef8f15f68f6928ff521334a9778ee9179df21243739d1e2490f"} err="failed to get container status \"99845b450fe48ef8f15f68f6928ff521334a9778ee9179df21243739d1e2490f\": rpc error: code = NotFound desc = could not find container \"99845b450fe48ef8f15f68f6928ff521334a9778ee9179df21243739d1e2490f\": container with ID starting with 99845b450fe48ef8f15f68f6928ff521334a9778ee9179df21243739d1e2490f not found: ID does not exist" Oct 30 09:59:55.017185 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.017108 5938 scope.go:122] "RemoveContainer" containerID="8fba50c95fdca4091671a1c052d2a56fe007f6f558e474711dd72994e71272e7" Oct 30 09:59:55.026265 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.026250 5938 scope.go:122] "RemoveContainer" containerID="8fba50c95fdca4091671a1c052d2a56fe007f6f558e474711dd72994e71272e7" Oct 30 09:59:55.026533 ip-10-0-135-148 kubenswrapper[5938]: E1030 09:59:55.026515 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fba50c95fdca4091671a1c052d2a56fe007f6f558e474711dd72994e71272e7\": container with ID starting with 8fba50c95fdca4091671a1c052d2a56fe007f6f558e474711dd72994e71272e7 not found: ID does not exist" containerID="8fba50c95fdca4091671a1c052d2a56fe007f6f558e474711dd72994e71272e7" Oct 30 09:59:55.026627 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.026539 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fba50c95fdca4091671a1c052d2a56fe007f6f558e474711dd72994e71272e7"} err="failed to get container status \"8fba50c95fdca4091671a1c052d2a56fe007f6f558e474711dd72994e71272e7\": rpc error: code = NotFound desc = could not find container \"8fba50c95fdca4091671a1c052d2a56fe007f6f558e474711dd72994e71272e7\": container with ID starting with 8fba50c95fdca4091671a1c052d2a56fe007f6f558e474711dd72994e71272e7 not found: ID does not exist" Oct 30 09:59:55.061856 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.061833 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s"] Oct 30 09:59:55.065413 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.065393 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a479df-6869c8cd76-nd98s"] Oct 30 09:59:55.105456 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.105430 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l"] Oct 30 09:59:55.111102 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:55.111084 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a479df-64564c5b95-h7t4l"] Oct 30 09:59:56.256987 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:56.256956 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80bc4ca3-dd13-4fa4-900f-82b93b520ffe" path="/var/lib/kubelet/pods/80bc4ca3-dd13-4fa4-900f-82b93b520ffe/volumes" Oct 30 09:59:56.257406 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:56.257347 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c527fc-edd4-4a04-a781-483921b54c6b" path="/var/lib/kubelet/pods/e9c527fc-edd4-4a04-a781-483921b54c6b/volumes" Oct 30 09:59:57.621764 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:57.621731 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc"] Oct 30 09:59:57.622170 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:57.621917 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" podUID="02b49bb6-bb6c-4c24-9c9a-1fd74645757f" containerName="istio-proxy" containerID="cri-o://b684b1173841f3ea13bd6c000334fa8e479f1ecb2e6aa0bad135d7a5e32fddd8" gracePeriod=30 Oct 30 09:59:58.283195 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:58.283148 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh"] Oct 30 09:59:58.317455 ip-10-0-135-148 kubenswrapper[5938]: I1030 09:59:58.317427 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks"] Oct 30 10:00:00.142443 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.142404 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4"] Oct 30 10:00:00.142966 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.142947 5938 cpu_manager.go:414] "containerMap: removing container" podUID="dafca48f-8187-41c5-b23f-c9b600701701" containerName="revoke-keys" Oct 30 10:00:00.143045 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.142970 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafca48f-8187-41c5-b23f-c9b600701701" containerName="revoke-keys" Oct 30 10:00:00.143045 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.142981 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9cf4c90b-e480-41a5-9522-045ee3d94955" containerName="payload-processing" Oct 30 10:00:00.143045 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.142986 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cf4c90b-e480-41a5-9522-045ee3d94955" containerName="payload-processing" Oct 30 10:00:00.143045 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.142997 5938 cpu_manager.go:414] "containerMap: removing container" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" Oct 30 10:00:00.143045 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143002 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" Oct 30 10:00:00.143045 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143015 5938 cpu_manager.go:414] "containerMap: removing container" podUID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" containerName="main" Oct 30 10:00:00.143045 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143020 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" containerName="main" Oct 30 10:00:00.143045 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143027 5938 cpu_manager.go:414] "containerMap: removing container" podUID="80bc4ca3-dd13-4fa4-900f-82b93b520ffe" containerName="payload-processing" Oct 30 10:00:00.143045 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143033 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80bc4ca3-dd13-4fa4-900f-82b93b520ffe" containerName="payload-processing" Oct 30 10:00:00.143045 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143042 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2d4172ae-9665-4707-97ab-6b3c82fb7242" containerName="payload-processing" Oct 30 10:00:00.143045 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143048 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4172ae-9665-4707-97ab-6b3c82fb7242" containerName="payload-processing" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143054 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c7325b5f-f90d-44a9-876d-48245c0fe2b0" containerName="istio-proxy" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143059 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7325b5f-f90d-44a9-876d-48245c0fe2b0" containerName="istio-proxy" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143067 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e9c527fc-edd4-4a04-a781-483921b54c6b" containerName="payload-pre-processing" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143074 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c527fc-edd4-4a04-a781-483921b54c6b" containerName="payload-pre-processing" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143089 5938 cpu_manager.go:414] "containerMap: removing container" podUID="52f16458-2419-4f90-a9e4-1d2344e24509" containerName="main" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143094 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f16458-2419-4f90-a9e4-1d2344e24509" containerName="main" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143100 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2089f287-3340-45e9-99d4-9e3ce1fbbe9e" containerName="payload-processing" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143105 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2089f287-3340-45e9-99d4-9e3ce1fbbe9e" containerName="payload-processing" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143114 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5ba161e1-40bc-4ea2-858a-44b63d9bdc83" containerName="payload-pre-processing" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143118 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba161e1-40bc-4ea2-858a-44b63d9bdc83" containerName="payload-pre-processing" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143126 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143131 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143206 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2371dfe4-c1c9-4773-b650-4d28e86a2f08" containerName="main" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143218 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafca48f-8187-41c5-b23f-c9b600701701" containerName="revoke-keys" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143225 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cf4c90b-e480-41a5-9522-045ee3d94955" containerName="payload-processing" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143231 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="372a7ce3-4393-48e0-a192-10e0530e933d" containerName="main" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143237 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e9e5d6-3cf4-4d8b-90dc-f7920c8a1ed7" containerName="main" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143242 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="80bc4ca3-dd13-4fa4-900f-82b93b520ffe" containerName="payload-processing" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143248 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4172ae-9665-4707-97ab-6b3c82fb7242" containerName="payload-processing" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143253 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7325b5f-f90d-44a9-876d-48245c0fe2b0" containerName="istio-proxy" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143258 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c527fc-edd4-4a04-a781-483921b54c6b" containerName="payload-pre-processing" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143264 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f16458-2419-4f90-a9e4-1d2344e24509" containerName="main" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143270 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2089f287-3340-45e9-99d4-9e3ce1fbbe9e" containerName="payload-processing" Oct 30 10:00:00.143400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.143276 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba161e1-40bc-4ea2-858a-44b63d9bdc83" containerName="payload-pre-processing" Oct 30 10:00:00.148526 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.148504 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" Oct 30 10:00:00.170199 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.170171 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4"] Oct 30 10:00:00.188716 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.188682 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/067cbdb8-fb3e-422c-95ca-51d281a9dd76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4\" (UID: \"067cbdb8-fb3e-422c-95ca-51d281a9dd76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" Oct 30 10:00:00.188948 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.188922 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2xlz\" (UniqueName: \"kubernetes.io/projected/067cbdb8-fb3e-422c-95ca-51d281a9dd76-kube-api-access-x2xlz\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4\" (UID: \"067cbdb8-fb3e-422c-95ca-51d281a9dd76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" Oct 30 10:00:00.265752 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.265724 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 10:00:00.290128 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.290096 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/067cbdb8-fb3e-422c-95ca-51d281a9dd76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4\" (UID: \"067cbdb8-fb3e-422c-95ca-51d281a9dd76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" Oct 30 10:00:00.290320 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.290145 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2xlz\" (UniqueName: \"kubernetes.io/projected/067cbdb8-fb3e-422c-95ca-51d281a9dd76-kube-api-access-x2xlz\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4\" (UID: \"067cbdb8-fb3e-422c-95ca-51d281a9dd76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" Oct 30 10:00:00.290832 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.290810 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/067cbdb8-fb3e-422c-95ca-51d281a9dd76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4\" (UID: \"067cbdb8-fb3e-422c-95ca-51d281a9dd76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" Oct 30 10:00:00.311095 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.311074 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2xlz\" (UniqueName: \"kubernetes.io/projected/067cbdb8-fb3e-422c-95ca-51d281a9dd76-kube-api-access-x2xlz\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4\" (UID: \"067cbdb8-fb3e-422c-95ca-51d281a9dd76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" Oct 30 10:00:00.462350 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.462268 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" Oct 30 10:00:00.757972 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:00.757941 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4"] Oct 30 10:00:00.759291 ip-10-0-135-148 kubenswrapper[5938]: W1030 10:00:00.759262 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067cbdb8_fb3e_422c_95ca_51d281a9dd76.slice/crio-d37779b7a816f8c255bd6cc53ee39bcf5560fd89bc853522ddb72865d0c2aab4 WatchSource:0}: Error finding container d37779b7a816f8c255bd6cc53ee39bcf5560fd89bc853522ddb72865d0c2aab4: Status 404 returned error can't find the container with id d37779b7a816f8c255bd6cc53ee39bcf5560fd89bc853522ddb72865d0c2aab4 Oct 30 10:00:01.047163 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:01.047129 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" event={"ID":"067cbdb8-fb3e-422c-95ca-51d281a9dd76","Type":"ContainerStarted","Data":"456ba4218c0ffa7dda882e162a6308e2293e1f2bd19e04ab678a7313de6db633"} Oct 30 10:00:01.047284 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:01.047192 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" event={"ID":"067cbdb8-fb3e-422c-95ca-51d281a9dd76","Type":"ContainerStarted","Data":"d37779b7a816f8c255bd6cc53ee39bcf5560fd89bc853522ddb72865d0c2aab4"} Oct 30 10:00:01.118525 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:01.118479 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" podStartSLOduration=1.118465606 podStartE2EDuration="1.118465606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:00:01.11686241 +0000 UTC m=+3918.979568231" watchObservedRunningTime="2026-10-30 10:00:01.118465606 +0000 UTC m=+3918.981171427" Oct 30 10:00:02.054562 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:02.054526 5938 generic.go:356] "Generic (PLEG): container finished" podID="067cbdb8-fb3e-422c-95ca-51d281a9dd76" containerID="456ba4218c0ffa7dda882e162a6308e2293e1f2bd19e04ab678a7313de6db633" exitCode=60 Oct 30 10:00:02.055049 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:02.054624 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" event={"ID":"067cbdb8-fb3e-422c-95ca-51d281a9dd76","Type":"ContainerDied","Data":"456ba4218c0ffa7dda882e162a6308e2293e1f2bd19e04ab678a7313de6db633"} Oct 30 10:00:02.055049 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:02.054917 5938 scope.go:122] "RemoveContainer" containerID="456ba4218c0ffa7dda882e162a6308e2293e1f2bd19e04ab678a7313de6db633" Oct 30 10:00:02.999088 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:02.999063 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 10:00:03.063510 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.063480 5938 generic.go:356] "Generic (PLEG): container finished" podID="02b49bb6-bb6c-4c24-9c9a-1fd74645757f" containerID="b684b1173841f3ea13bd6c000334fa8e479f1ecb2e6aa0bad135d7a5e32fddd8" exitCode=0 Oct 30 10:00:03.064005 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.063539 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" Oct 30 10:00:03.064005 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.063567 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" event={"ID":"02b49bb6-bb6c-4c24-9c9a-1fd74645757f","Type":"ContainerDied","Data":"b684b1173841f3ea13bd6c000334fa8e479f1ecb2e6aa0bad135d7a5e32fddd8"} Oct 30 10:00:03.064005 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.063650 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc" event={"ID":"02b49bb6-bb6c-4c24-9c9a-1fd74645757f","Type":"ContainerDied","Data":"321cd4d584bb85f573c442c5f95ff95b7b64c2795d9fd206a3e354cb819815e8"} Oct 30 10:00:03.064005 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.063681 5938 scope.go:122] "RemoveContainer" containerID="b684b1173841f3ea13bd6c000334fa8e479f1ecb2e6aa0bad135d7a5e32fddd8" Oct 30 10:00:03.067452 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.066414 5938 generic.go:356] "Generic (PLEG): container finished" podID="067cbdb8-fb3e-422c-95ca-51d281a9dd76" containerID="96e8d1e73bb88da5d1180d2a832ca045cb41cfafec562823f65ab8c79bb7a17f" exitCode=60 Oct 30 10:00:03.067452 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.066487 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" event={"ID":"067cbdb8-fb3e-422c-95ca-51d281a9dd76","Type":"ContainerDied","Data":"96e8d1e73bb88da5d1180d2a832ca045cb41cfafec562823f65ab8c79bb7a17f"} Oct 30 10:00:03.067452 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.066913 5938 scope.go:122] "RemoveContainer" containerID="96e8d1e73bb88da5d1180d2a832ca045cb41cfafec562823f65ab8c79bb7a17f" Oct 30 10:00:03.067452 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:00:03.067224 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4_odh-ai-gateway-infra(067cbdb8-fb3e-422c-95ca-51d281a9dd76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" podUID="067cbdb8-fb3e-422c-95ca-51d281a9dd76" Oct 30 10:00:03.078846 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.078819 5938 scope.go:122] "RemoveContainer" containerID="b684b1173841f3ea13bd6c000334fa8e479f1ecb2e6aa0bad135d7a5e32fddd8" Oct 30 10:00:03.079264 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:00:03.079235 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b684b1173841f3ea13bd6c000334fa8e479f1ecb2e6aa0bad135d7a5e32fddd8\": container with ID starting with b684b1173841f3ea13bd6c000334fa8e479f1ecb2e6aa0bad135d7a5e32fddd8 not found: ID does not exist" containerID="b684b1173841f3ea13bd6c000334fa8e479f1ecb2e6aa0bad135d7a5e32fddd8" Oct 30 10:00:03.079362 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.079278 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b684b1173841f3ea13bd6c000334fa8e479f1ecb2e6aa0bad135d7a5e32fddd8"} err="failed to get container status \"b684b1173841f3ea13bd6c000334fa8e479f1ecb2e6aa0bad135d7a5e32fddd8\": rpc error: code = NotFound desc = could not find container \"b684b1173841f3ea13bd6c000334fa8e479f1ecb2e6aa0bad135d7a5e32fddd8\": container with ID starting with b684b1173841f3ea13bd6c000334fa8e479f1ecb2e6aa0bad135d7a5e32fddd8 not found: ID does not exist" Oct 30 10:00:03.079362 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.079302 5938 scope.go:122] "RemoveContainer" containerID="456ba4218c0ffa7dda882e162a6308e2293e1f2bd19e04ab678a7313de6db633" Oct 30 10:00:03.118923 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.118893 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-kube-api-access-dxqmq\" (UniqueName: \"kubernetes.io/projected/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-kube-api-access-dxqmq\") pod \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " Oct 30 10:00:03.118923 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.118938 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-workload-socket\") pod \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " Oct 30 10:00:03.119553 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.118985 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-workload-certs\") pod \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " Oct 30 10:00:03.119553 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.119017 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istiod-ca-cert\") pod \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " Oct 30 10:00:03.119553 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.119047 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-podinfo\") pod \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " Oct 30 10:00:03.119553 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.119121 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-envoy\") pod \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " Oct 30 10:00:03.119553 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.119181 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-token\" (UniqueName: \"kubernetes.io/projected/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-token\") pod \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " Oct 30 10:00:03.119553 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.119216 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-credential-socket\") pod \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " Oct 30 10:00:03.119553 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.119246 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-data\") pod \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\" (UID: \"02b49bb6-bb6c-4c24-9c9a-1fd74645757f\") " Oct 30 10:00:03.119553 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.119286 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-workload-socket" pod "02b49bb6-bb6c-4c24-9c9a-1fd74645757f" (UID: "02b49bb6-bb6c-4c24-9c9a-1fd74645757f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:03.119553 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.119304 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-workload-certs" pod "02b49bb6-bb6c-4c24-9c9a-1fd74645757f" (UID: "02b49bb6-bb6c-4c24-9c9a-1fd74645757f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:03.120114 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.119822 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-data" pod "02b49bb6-bb6c-4c24-9c9a-1fd74645757f" (UID: "02b49bb6-bb6c-4c24-9c9a-1fd74645757f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:03.120114 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.120054 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istiod-ca-cert" pod "02b49bb6-bb6c-4c24-9c9a-1fd74645757f" (UID: "02b49bb6-bb6c-4c24-9c9a-1fd74645757f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:03.120242 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.120197 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-credential-socket" pod "02b49bb6-bb6c-4c24-9c9a-1fd74645757f" (UID: "02b49bb6-bb6c-4c24-9c9a-1fd74645757f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:03.120242 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.120211 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:03.120728 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.120477 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:03.122439 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.122417 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-token" pod "02b49bb6-bb6c-4c24-9c9a-1fd74645757f" (UID: "02b49bb6-bb6c-4c24-9c9a-1fd74645757f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:03.122636 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.122610 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-envoy" pod "02b49bb6-bb6c-4c24-9c9a-1fd74645757f" (UID: "02b49bb6-bb6c-4c24-9c9a-1fd74645757f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:03.122636 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.122623 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-kube-api-access-dxqmq" pod "02b49bb6-bb6c-4c24-9c9a-1fd74645757f" (UID: "02b49bb6-bb6c-4c24-9c9a-1fd74645757f"). InnerVolumeSpecName "kube-api-access-dxqmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:03.123231 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.123202 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-podinfo" pod "02b49bb6-bb6c-4c24-9c9a-1fd74645757f" (UID: "02b49bb6-bb6c-4c24-9c9a-1fd74645757f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 10:00:03.221184 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.221117 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:03.221184 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.221141 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:03.221184 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.221150 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:03.221184 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.221161 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:03.221184 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.221170 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:03.221184 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.221179 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:03.221184 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.221187 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxqmq\" (UniqueName: \"kubernetes.io/projected/02b49bb6-bb6c-4c24-9c9a-1fd74645757f-kube-api-access-dxqmq\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:03.283604 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.283552 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" podUID="75d0141a-4aa4-41fe-b47d-1e564cc52302" containerName="payload-processing" containerID="cri-o://ca0bc92ce1d02eabd7d874e83ab85ebea78e16e43bc38589b4b3dab4ae500f78" gracePeriod=25 Oct 30 10:00:03.317977 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.317932 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" podUID="7a880792-dae1-47f4-96e9-7df47ce563cf" containerName="payload-pre-processing" containerID="cri-o://bc4f67bcbfa99fa4056c91f24b995633800803385e7c0ffa3bfe5d100ced17d0" gracePeriod=25 Oct 30 10:00:03.438477 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.438447 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc"] Oct 30 10:00:03.442793 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.442767 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hxpzc"] Oct 30 10:00:03.548190 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.548166 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" Oct 30 10:00:03.578805 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.578777 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp"] Oct 30 10:00:03.579021 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.578974 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" podUID="457441a6-59df-4b33-8675-55f9b5382fad" containerName="maas-api" containerID="cri-o://5bc7ca94fdbcefe4190bdd242b7f5294bc0c0d0484836c7c4b1e936283821adc" gracePeriod=30 Oct 30 10:00:03.624706 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.624673 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d0141a-4aa4-41fe-b47d-1e564cc52302-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d0141a-4aa4-41fe-b47d-1e564cc52302-plugins-config-volume\") pod \"75d0141a-4aa4-41fe-b47d-1e564cc52302\" (UID: \"75d0141a-4aa4-41fe-b47d-1e564cc52302\") " Oct 30 10:00:03.624949 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.624929 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d0141a-4aa4-41fe-b47d-1e564cc52302-kube-api-access-dbmb8\" (UniqueName: \"kubernetes.io/projected/75d0141a-4aa4-41fe-b47d-1e564cc52302-kube-api-access-dbmb8\") pod \"75d0141a-4aa4-41fe-b47d-1e564cc52302\" (UID: \"75d0141a-4aa4-41fe-b47d-1e564cc52302\") " Oct 30 10:00:03.625109 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.625081 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d0141a-4aa4-41fe-b47d-1e564cc52302-plugins-config-volume" pod "75d0141a-4aa4-41fe-b47d-1e564cc52302" (UID: "75d0141a-4aa4-41fe-b47d-1e564cc52302"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:03.625413 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.625379 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d0141a-4aa4-41fe-b47d-1e564cc52302-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:03.627464 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.627437 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d0141a-4aa4-41fe-b47d-1e564cc52302-kube-api-access-dbmb8" pod "75d0141a-4aa4-41fe-b47d-1e564cc52302" (UID: "75d0141a-4aa4-41fe-b47d-1e564cc52302"). InnerVolumeSpecName "kube-api-access-dbmb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:03.713578 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.713548 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" Oct 30 10:00:03.726258 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.726231 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbmb8\" (UniqueName: \"kubernetes.io/projected/75d0141a-4aa4-41fe-b47d-1e564cc52302-kube-api-access-dbmb8\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:03.828398 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.828362 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a880792-dae1-47f4-96e9-7df47ce563cf-kube-api-access-k2254\" (UniqueName: \"kubernetes.io/projected/7a880792-dae1-47f4-96e9-7df47ce563cf-kube-api-access-k2254\") pod \"7a880792-dae1-47f4-96e9-7df47ce563cf\" (UID: \"7a880792-dae1-47f4-96e9-7df47ce563cf\") " Oct 30 10:00:03.828793 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.828771 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a880792-dae1-47f4-96e9-7df47ce563cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a880792-dae1-47f4-96e9-7df47ce563cf-plugins-config-volume\") pod \"7a880792-dae1-47f4-96e9-7df47ce563cf\" (UID: \"7a880792-dae1-47f4-96e9-7df47ce563cf\") " Oct 30 10:00:03.829212 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.829188 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a880792-dae1-47f4-96e9-7df47ce563cf-plugins-config-volume" pod "7a880792-dae1-47f4-96e9-7df47ce563cf" (UID: "7a880792-dae1-47f4-96e9-7df47ce563cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:03.829417 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.829395 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a880792-dae1-47f4-96e9-7df47ce563cf-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:03.833443 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.832235 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a880792-dae1-47f4-96e9-7df47ce563cf-kube-api-access-k2254" pod "7a880792-dae1-47f4-96e9-7df47ce563cf" (UID: "7a880792-dae1-47f4-96e9-7df47ce563cf"). InnerVolumeSpecName "kube-api-access-k2254". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:03.861826 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.861800 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 10:00:03.930300 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.930265 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/457441a6-59df-4b33-8675-55f9b5382fad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457441a6-59df-4b33-8675-55f9b5382fad-maas-api-tls\") pod \"457441a6-59df-4b33-8675-55f9b5382fad\" (UID: \"457441a6-59df-4b33-8675-55f9b5382fad\") " Oct 30 10:00:03.930484 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.930343 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/457441a6-59df-4b33-8675-55f9b5382fad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457441a6-59df-4b33-8675-55f9b5382fad-openshift-service-ca\") pod \"457441a6-59df-4b33-8675-55f9b5382fad\" (UID: \"457441a6-59df-4b33-8675-55f9b5382fad\") " Oct 30 10:00:03.930557 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.930503 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/457441a6-59df-4b33-8675-55f9b5382fad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/457441a6-59df-4b33-8675-55f9b5382fad-metrics-certs\") pod \"457441a6-59df-4b33-8675-55f9b5382fad\" (UID: \"457441a6-59df-4b33-8675-55f9b5382fad\") " Oct 30 10:00:03.930557 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.930549 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457441a6-59df-4b33-8675-55f9b5382fad-kube-api-access-qd2mz\" (UniqueName: \"kubernetes.io/projected/457441a6-59df-4b33-8675-55f9b5382fad-kube-api-access-qd2mz\") pod \"457441a6-59df-4b33-8675-55f9b5382fad\" (UID: \"457441a6-59df-4b33-8675-55f9b5382fad\") " Oct 30 10:00:03.930781 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.930760 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457441a6-59df-4b33-8675-55f9b5382fad-openshift-service-ca" pod "457441a6-59df-4b33-8675-55f9b5382fad" (UID: "457441a6-59df-4b33-8675-55f9b5382fad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:03.930972 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.930954 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2254\" (UniqueName: \"kubernetes.io/projected/7a880792-dae1-47f4-96e9-7df47ce563cf-kube-api-access-k2254\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:03.931031 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.930976 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457441a6-59df-4b33-8675-55f9b5382fad-openshift-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:03.932726 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.932703 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457441a6-59df-4b33-8675-55f9b5382fad-metrics-certs" pod "457441a6-59df-4b33-8675-55f9b5382fad" (UID: "457441a6-59df-4b33-8675-55f9b5382fad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:00:03.932820 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.932737 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457441a6-59df-4b33-8675-55f9b5382fad-maas-api-tls" pod "457441a6-59df-4b33-8675-55f9b5382fad" (UID: "457441a6-59df-4b33-8675-55f9b5382fad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:00:03.933123 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:03.933103 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457441a6-59df-4b33-8675-55f9b5382fad-kube-api-access-qd2mz" pod "457441a6-59df-4b33-8675-55f9b5382fad" (UID: "457441a6-59df-4b33-8675-55f9b5382fad"). InnerVolumeSpecName "kube-api-access-qd2mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:04.031984 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.031902 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457441a6-59df-4b33-8675-55f9b5382fad-metrics-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:04.031984 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.031929 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd2mz\" (UniqueName: \"kubernetes.io/projected/457441a6-59df-4b33-8675-55f9b5382fad-kube-api-access-qd2mz\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:04.031984 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.031940 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457441a6-59df-4b33-8675-55f9b5382fad-maas-api-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:04.075216 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.075178 5938 generic.go:356] "Generic (PLEG): container finished" podID="75d0141a-4aa4-41fe-b47d-1e564cc52302" containerID="ca0bc92ce1d02eabd7d874e83ab85ebea78e16e43bc38589b4b3dab4ae500f78" exitCode=0 Oct 30 10:00:04.075722 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.075227 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" Oct 30 10:00:04.075722 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.075260 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" event={"ID":"75d0141a-4aa4-41fe-b47d-1e564cc52302","Type":"ContainerDied","Data":"ca0bc92ce1d02eabd7d874e83ab85ebea78e16e43bc38589b4b3dab4ae500f78"} Oct 30 10:00:04.075722 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.075312 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh" event={"ID":"75d0141a-4aa4-41fe-b47d-1e564cc52302","Type":"ContainerDied","Data":"96476d8bd403f328e0fc0409da650a5d41feaa280ec4b364e28130c1eec1732c"} Oct 30 10:00:04.075722 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.075334 5938 scope.go:122] "RemoveContainer" containerID="ca0bc92ce1d02eabd7d874e83ab85ebea78e16e43bc38589b4b3dab4ae500f78" Oct 30 10:00:04.078072 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.078049 5938 generic.go:356] "Generic (PLEG): container finished" podID="7a880792-dae1-47f4-96e9-7df47ce563cf" containerID="bc4f67bcbfa99fa4056c91f24b995633800803385e7c0ffa3bfe5d100ced17d0" exitCode=0 Oct 30 10:00:04.078221 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.078108 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" Oct 30 10:00:04.078221 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.078125 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" event={"ID":"7a880792-dae1-47f4-96e9-7df47ce563cf","Type":"ContainerDied","Data":"bc4f67bcbfa99fa4056c91f24b995633800803385e7c0ffa3bfe5d100ced17d0"} Oct 30 10:00:04.078221 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.078186 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" event={"ID":"7a880792-dae1-47f4-96e9-7df47ce563cf","Type":"ContainerDied","Data":"e710c2bee8fa438779d9e188ce779200ec5c46206ab29f8c17224b77ec47a79d"} Oct 30 10:00:04.080661 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.080638 5938 scope.go:122] "RemoveContainer" containerID="96e8d1e73bb88da5d1180d2a832ca045cb41cfafec562823f65ab8c79bb7a17f" Oct 30 10:00:04.080906 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:00:04.080884 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4_odh-ai-gateway-infra(067cbdb8-fb3e-422c-95ca-51d281a9dd76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" podUID="067cbdb8-fb3e-422c-95ca-51d281a9dd76" Oct 30 10:00:04.081970 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.081947 5938 generic.go:356] "Generic (PLEG): container finished" podID="457441a6-59df-4b33-8675-55f9b5382fad" containerID="5bc7ca94fdbcefe4190bdd242b7f5294bc0c0d0484836c7c4b1e936283821adc" exitCode=0 Oct 30 10:00:04.082081 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.081997 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" Oct 30 10:00:04.082081 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.082007 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" event={"ID":"457441a6-59df-4b33-8675-55f9b5382fad","Type":"ContainerDied","Data":"5bc7ca94fdbcefe4190bdd242b7f5294bc0c0d0484836c7c4b1e936283821adc"} Oct 30 10:00:04.082081 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.082048 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp" event={"ID":"457441a6-59df-4b33-8675-55f9b5382fad","Type":"ContainerDied","Data":"c7f7a463e96a55923594237494aae8cb6e20e1f98aa0dcda5026e69389c10638"} Oct 30 10:00:04.088754 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.088665 5938 scope.go:122] "RemoveContainer" containerID="ca0bc92ce1d02eabd7d874e83ab85ebea78e16e43bc38589b4b3dab4ae500f78" Oct 30 10:00:04.089099 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:00:04.088998 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0bc92ce1d02eabd7d874e83ab85ebea78e16e43bc38589b4b3dab4ae500f78\": container with ID starting with ca0bc92ce1d02eabd7d874e83ab85ebea78e16e43bc38589b4b3dab4ae500f78 not found: ID does not exist" containerID="ca0bc92ce1d02eabd7d874e83ab85ebea78e16e43bc38589b4b3dab4ae500f78" Oct 30 10:00:04.089099 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.089034 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0bc92ce1d02eabd7d874e83ab85ebea78e16e43bc38589b4b3dab4ae500f78"} err="failed to get container status \"ca0bc92ce1d02eabd7d874e83ab85ebea78e16e43bc38589b4b3dab4ae500f78\": rpc error: code = NotFound desc = could not find container \"ca0bc92ce1d02eabd7d874e83ab85ebea78e16e43bc38589b4b3dab4ae500f78\": container with ID starting with ca0bc92ce1d02eabd7d874e83ab85ebea78e16e43bc38589b4b3dab4ae500f78 not found: ID does not exist" Oct 30 10:00:04.089099 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.089055 5938 scope.go:122] "RemoveContainer" containerID="bc4f67bcbfa99fa4056c91f24b995633800803385e7c0ffa3bfe5d100ced17d0" Oct 30 10:00:04.100541 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.100521 5938 scope.go:122] "RemoveContainer" containerID="bc4f67bcbfa99fa4056c91f24b995633800803385e7c0ffa3bfe5d100ced17d0" Oct 30 10:00:04.100855 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:00:04.100835 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4f67bcbfa99fa4056c91f24b995633800803385e7c0ffa3bfe5d100ced17d0\": container with ID starting with bc4f67bcbfa99fa4056c91f24b995633800803385e7c0ffa3bfe5d100ced17d0 not found: ID does not exist" containerID="bc4f67bcbfa99fa4056c91f24b995633800803385e7c0ffa3bfe5d100ced17d0" Oct 30 10:00:04.100936 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.100862 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4f67bcbfa99fa4056c91f24b995633800803385e7c0ffa3bfe5d100ced17d0"} err="failed to get container status \"bc4f67bcbfa99fa4056c91f24b995633800803385e7c0ffa3bfe5d100ced17d0\": rpc error: code = NotFound desc = could not find container \"bc4f67bcbfa99fa4056c91f24b995633800803385e7c0ffa3bfe5d100ced17d0\": container with ID starting with bc4f67bcbfa99fa4056c91f24b995633800803385e7c0ffa3bfe5d100ced17d0 not found: ID does not exist" Oct 30 10:00:04.100936 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.100877 5938 scope.go:122] "RemoveContainer" containerID="5bc7ca94fdbcefe4190bdd242b7f5294bc0c0d0484836c7c4b1e936283821adc" Oct 30 10:00:04.110451 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.110432 5938 scope.go:122] "RemoveContainer" containerID="5bc7ca94fdbcefe4190bdd242b7f5294bc0c0d0484836c7c4b1e936283821adc" Oct 30 10:00:04.110779 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:00:04.110748 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc7ca94fdbcefe4190bdd242b7f5294bc0c0d0484836c7c4b1e936283821adc\": container with ID starting with 5bc7ca94fdbcefe4190bdd242b7f5294bc0c0d0484836c7c4b1e936283821adc not found: ID does not exist" containerID="5bc7ca94fdbcefe4190bdd242b7f5294bc0c0d0484836c7c4b1e936283821adc" Oct 30 10:00:04.110856 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.110790 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc7ca94fdbcefe4190bdd242b7f5294bc0c0d0484836c7c4b1e936283821adc"} err="failed to get container status \"5bc7ca94fdbcefe4190bdd242b7f5294bc0c0d0484836c7c4b1e936283821adc\": rpc error: code = NotFound desc = could not find container \"5bc7ca94fdbcefe4190bdd242b7f5294bc0c0d0484836c7c4b1e936283821adc\": container with ID starting with 5bc7ca94fdbcefe4190bdd242b7f5294bc0c0d0484836c7c4b1e936283821adc not found: ID does not exist" Oct 30 10:00:04.171660 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.171619 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks"] Oct 30 10:00:04.174990 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.174965 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks"] Oct 30 10:00:04.212650 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.212626 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh"] Oct 30 10:00:04.218814 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.218792 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a525bd-5ff45ff7fd-vmtbh"] Oct 30 10:00:04.264692 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.264654 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b49bb6-bb6c-4c24-9c9a-1fd74645757f" path="/var/lib/kubelet/pods/02b49bb6-bb6c-4c24-9c9a-1fd74645757f/volumes" Oct 30 10:00:04.265190 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.265171 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d0141a-4aa4-41fe-b47d-1e564cc52302" path="/var/lib/kubelet/pods/75d0141a-4aa4-41fe-b47d-1e564cc52302/volumes" Oct 30 10:00:04.265662 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.265641 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a880792-dae1-47f4-96e9-7df47ce563cf" path="/var/lib/kubelet/pods/7a880792-dae1-47f4-96e9-7df47ce563cf/volumes" Oct 30 10:00:04.272654 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.272631 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp"] Oct 30 10:00:04.276274 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.276251 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3a61e1-bf857dff-qsvgp"] Oct 30 10:00:04.667796 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:04.667756 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a525bd-76c6469fff-stjks" podUID="7a880792-dae1-47f4-96e9-7df47ce563cf" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.153:50052\" within 1s: context deadline exceeded" Oct 30 10:00:04.667956 ip-10-0-135-148 kubenswrapper[5938]: W1030 10:00:04.667874 5938 logging.go:55] [core] [Channel #8059 SubChannel #8060]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.153:50052", ServerName: "10.134.0.153:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.153:50052: operation was canceled" Oct 30 10:00:05.663351 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:05.663320 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s"] Oct 30 10:00:05.663843 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:05.663561 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" podUID="665293f1-a5e7-46a9-a797-9fee3027b9ba" containerName="maas-api" containerID="cri-o://5bf44ce12094f98464dd3f388135ca3efab01e51a0fd1b54569ad907a1eeca52" gracePeriod=30 Oct 30 10:00:05.939997 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:05.939976 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 10:00:06.055965 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.055935 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/665293f1-a5e7-46a9-a797-9fee3027b9ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/665293f1-a5e7-46a9-a797-9fee3027b9ba-openshift-service-ca\") pod \"665293f1-a5e7-46a9-a797-9fee3027b9ba\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " Oct 30 10:00:06.056149 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.056087 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-metrics-certs\") pod \"665293f1-a5e7-46a9-a797-9fee3027b9ba\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " Oct 30 10:00:06.056149 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.056110 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-maas-api-tls\") pod \"665293f1-a5e7-46a9-a797-9fee3027b9ba\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " Oct 30 10:00:06.056149 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.056131 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/665293f1-a5e7-46a9-a797-9fee3027b9ba-kube-api-access-rngj2\" (UniqueName: \"kubernetes.io/projected/665293f1-a5e7-46a9-a797-9fee3027b9ba-kube-api-access-rngj2\") pod \"665293f1-a5e7-46a9-a797-9fee3027b9ba\" (UID: \"665293f1-a5e7-46a9-a797-9fee3027b9ba\") " Oct 30 10:00:06.056347 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.056319 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/665293f1-a5e7-46a9-a797-9fee3027b9ba-openshift-service-ca" pod "665293f1-a5e7-46a9-a797-9fee3027b9ba" (UID: "665293f1-a5e7-46a9-a797-9fee3027b9ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:06.056545 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.056526 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/665293f1-a5e7-46a9-a797-9fee3027b9ba-openshift-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:06.058429 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.058400 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-maas-api-tls" pod "665293f1-a5e7-46a9-a797-9fee3027b9ba" (UID: "665293f1-a5e7-46a9-a797-9fee3027b9ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:00:06.058429 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.058417 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-metrics-certs" pod "665293f1-a5e7-46a9-a797-9fee3027b9ba" (UID: "665293f1-a5e7-46a9-a797-9fee3027b9ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:00:06.058641 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.058469 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665293f1-a5e7-46a9-a797-9fee3027b9ba-kube-api-access-rngj2" pod "665293f1-a5e7-46a9-a797-9fee3027b9ba" (UID: "665293f1-a5e7-46a9-a797-9fee3027b9ba"). InnerVolumeSpecName "kube-api-access-rngj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:06.101281 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.101245 5938 generic.go:356] "Generic (PLEG): container finished" podID="665293f1-a5e7-46a9-a797-9fee3027b9ba" containerID="5bf44ce12094f98464dd3f388135ca3efab01e51a0fd1b54569ad907a1eeca52" exitCode=0 Oct 30 10:00:06.101993 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.101294 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" Oct 30 10:00:06.101993 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.101341 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" event={"ID":"665293f1-a5e7-46a9-a797-9fee3027b9ba","Type":"ContainerDied","Data":"5bf44ce12094f98464dd3f388135ca3efab01e51a0fd1b54569ad907a1eeca52"} Oct 30 10:00:06.101993 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.101397 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s" event={"ID":"665293f1-a5e7-46a9-a797-9fee3027b9ba","Type":"ContainerDied","Data":"8c807fda1387954c809776dd44f6da24cb8a78550e2237222e7678f3cc08a028"} Oct 30 10:00:06.101993 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.101429 5938 scope.go:122] "RemoveContainer" containerID="5bf44ce12094f98464dd3f388135ca3efab01e51a0fd1b54569ad907a1eeca52" Oct 30 10:00:06.114332 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.114185 5938 scope.go:122] "RemoveContainer" containerID="5bf44ce12094f98464dd3f388135ca3efab01e51a0fd1b54569ad907a1eeca52" Oct 30 10:00:06.118447 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:00:06.118393 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf44ce12094f98464dd3f388135ca3efab01e51a0fd1b54569ad907a1eeca52\": container with ID starting with 5bf44ce12094f98464dd3f388135ca3efab01e51a0fd1b54569ad907a1eeca52 not found: ID does not exist" containerID="5bf44ce12094f98464dd3f388135ca3efab01e51a0fd1b54569ad907a1eeca52" Oct 30 10:00:06.118575 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.118437 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf44ce12094f98464dd3f388135ca3efab01e51a0fd1b54569ad907a1eeca52"} err="failed to get container status \"5bf44ce12094f98464dd3f388135ca3efab01e51a0fd1b54569ad907a1eeca52\": rpc error: code = NotFound desc = could not find container \"5bf44ce12094f98464dd3f388135ca3efab01e51a0fd1b54569ad907a1eeca52\": container with ID starting with 5bf44ce12094f98464dd3f388135ca3efab01e51a0fd1b54569ad907a1eeca52 not found: ID does not exist" Oct 30 10:00:06.156852 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.156810 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s"] Oct 30 10:00:06.157132 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.157111 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-metrics-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:06.157174 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.157141 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665293f1-a5e7-46a9-a797-9fee3027b9ba-maas-api-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:06.157174 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.157155 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rngj2\" (UniqueName: \"kubernetes.io/projected/665293f1-a5e7-46a9-a797-9fee3027b9ba-kube-api-access-rngj2\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:06.162575 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.162551 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a525bd-68cdccf4db-82g2s"] Oct 30 10:00:06.256798 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.256728 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457441a6-59df-4b33-8675-55f9b5382fad" path="/var/lib/kubelet/pods/457441a6-59df-4b33-8675-55f9b5382fad/volumes" Oct 30 10:00:06.257149 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:06.257136 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="665293f1-a5e7-46a9-a797-9fee3027b9ba" path="/var/lib/kubelet/pods/665293f1-a5e7-46a9-a797-9fee3027b9ba/volumes" Oct 30 10:00:12.838226 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:12.838190 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt"] Oct 30 10:00:12.838754 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:12.838451 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" podUID="f2b81448-0b6f-4093-9bdc-37139355b7e1" containerName="istio-proxy" containerID="cri-o://3872aa546cdf36a12365fd99d37eb06528a6b8607721d89e8925211c8e5b60c9" gracePeriod=30 Oct 30 10:00:14.187778 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.187747 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp"] Oct 30 10:00:14.188158 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.188145 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7a880792-dae1-47f4-96e9-7df47ce563cf" containerName="payload-pre-processing" Oct 30 10:00:14.188199 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.188159 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a880792-dae1-47f4-96e9-7df47ce563cf" containerName="payload-pre-processing" Oct 30 10:00:14.188199 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.188177 5938 cpu_manager.go:414] "containerMap: removing container" podUID="02b49bb6-bb6c-4c24-9c9a-1fd74645757f" containerName="istio-proxy" Oct 30 10:00:14.188199 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.188183 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b49bb6-bb6c-4c24-9c9a-1fd74645757f" containerName="istio-proxy" Oct 30 10:00:14.188199 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.188199 5938 cpu_manager.go:414] "containerMap: removing container" podUID="457441a6-59df-4b33-8675-55f9b5382fad" containerName="maas-api" Oct 30 10:00:14.188340 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.188204 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457441a6-59df-4b33-8675-55f9b5382fad" containerName="maas-api" Oct 30 10:00:14.188340 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.188211 5938 cpu_manager.go:414] "containerMap: removing container" podUID="665293f1-a5e7-46a9-a797-9fee3027b9ba" containerName="maas-api" Oct 30 10:00:14.188340 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.188215 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="665293f1-a5e7-46a9-a797-9fee3027b9ba" containerName="maas-api" Oct 30 10:00:14.188340 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.188233 5938 cpu_manager.go:414] "containerMap: removing container" podUID="75d0141a-4aa4-41fe-b47d-1e564cc52302" containerName="payload-processing" Oct 30 10:00:14.188340 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.188238 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d0141a-4aa4-41fe-b47d-1e564cc52302" containerName="payload-processing" Oct 30 10:00:14.188340 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.188290 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a880792-dae1-47f4-96e9-7df47ce563cf" containerName="payload-pre-processing" Oct 30 10:00:14.188340 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.188302 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b49bb6-bb6c-4c24-9c9a-1fd74645757f" containerName="istio-proxy" Oct 30 10:00:14.188340 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.188310 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="457441a6-59df-4b33-8675-55f9b5382fad" containerName="maas-api" Oct 30 10:00:14.188340 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.188315 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="665293f1-a5e7-46a9-a797-9fee3027b9ba" containerName="maas-api" Oct 30 10:00:14.188340 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.188323 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d0141a-4aa4-41fe-b47d-1e564cc52302" containerName="payload-processing" Oct 30 10:00:14.192869 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.192850 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" Oct 30 10:00:14.208922 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.208901 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp"] Oct 30 10:00:14.271050 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.271020 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 10:00:14.340252 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.340223 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d4f8d1-4871-4e09-a254-79262a3b0482-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp\" (UID: \"f9d4f8d1-4871-4e09-a254-79262a3b0482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" Oct 30 10:00:14.340386 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.340270 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d4f8d1-4871-4e09-a254-79262a3b0482-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp\" (UID: \"f9d4f8d1-4871-4e09-a254-79262a3b0482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" Oct 30 10:00:14.340437 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.340404 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vk5f\" (UniqueName: \"kubernetes.io/projected/f9d4f8d1-4871-4e09-a254-79262a3b0482-kube-api-access-9vk5f\") pod \"maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp\" (UID: \"f9d4f8d1-4871-4e09-a254-79262a3b0482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" Oct 30 10:00:14.442002 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.441920 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vk5f\" (UniqueName: \"kubernetes.io/projected/f9d4f8d1-4871-4e09-a254-79262a3b0482-kube-api-access-9vk5f\") pod \"maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp\" (UID: \"f9d4f8d1-4871-4e09-a254-79262a3b0482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" Oct 30 10:00:14.442002 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.441984 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d4f8d1-4871-4e09-a254-79262a3b0482-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp\" (UID: \"f9d4f8d1-4871-4e09-a254-79262a3b0482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" Oct 30 10:00:14.442251 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.442020 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d4f8d1-4871-4e09-a254-79262a3b0482-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp\" (UID: \"f9d4f8d1-4871-4e09-a254-79262a3b0482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" Oct 30 10:00:14.442688 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.442659 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d4f8d1-4871-4e09-a254-79262a3b0482-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp\" (UID: \"f9d4f8d1-4871-4e09-a254-79262a3b0482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" Oct 30 10:00:14.444602 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.444572 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d4f8d1-4871-4e09-a254-79262a3b0482-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp\" (UID: \"f9d4f8d1-4871-4e09-a254-79262a3b0482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" Oct 30 10:00:14.462566 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.462544 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vk5f\" (UniqueName: \"kubernetes.io/projected/f9d4f8d1-4871-4e09-a254-79262a3b0482-kube-api-access-9vk5f\") pod \"maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp\" (UID: \"f9d4f8d1-4871-4e09-a254-79262a3b0482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" Oct 30 10:00:14.505553 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.505531 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" Oct 30 10:00:14.807647 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:14.807383 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp"] Oct 30 10:00:14.810437 ip-10-0-135-148 kubenswrapper[5938]: W1030 10:00:14.810407 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d4f8d1_4871_4e09_a254_79262a3b0482.slice/crio-a6ad8d203743ea1e01c57549cc0f4a4950098010584a6602f618ff0cb28c0e5d WatchSource:0}: Error finding container a6ad8d203743ea1e01c57549cc0f4a4950098010584a6602f618ff0cb28c0e5d: Status 404 returned error can't find the container with id a6ad8d203743ea1e01c57549cc0f4a4950098010584a6602f618ff0cb28c0e5d Oct 30 10:00:15.159213 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:15.159183 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" event={"ID":"f9d4f8d1-4871-4e09-a254-79262a3b0482","Type":"ContainerStarted","Data":"7050af93cd46190b6402faa05f4eb4125521053cd5281ff4ae488466fcb3d276"} Oct 30 10:00:15.159327 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:15.159240 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" event={"ID":"f9d4f8d1-4871-4e09-a254-79262a3b0482","Type":"ContainerStarted","Data":"a6ad8d203743ea1e01c57549cc0f4a4950098010584a6602f618ff0cb28c0e5d"} Oct 30 10:00:15.200817 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:15.200772 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" podStartSLOduration=1.200755063 podStartE2EDuration="1.200755063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:00:15.200577219 +0000 UTC m=+3933.063283039" watchObservedRunningTime="2026-10-30 10:00:15.200755063 +0000 UTC m=+3933.063460895" Oct 30 10:00:15.808601 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:00:15.808548 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 10:00:16.167290 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:16.167253 5938 generic.go:356] "Generic (PLEG): container finished" podID="f9d4f8d1-4871-4e09-a254-79262a3b0482" containerID="7050af93cd46190b6402faa05f4eb4125521053cd5281ff4ae488466fcb3d276" exitCode=0 Oct 30 10:00:16.167470 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:16.167341 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" event={"ID":"f9d4f8d1-4871-4e09-a254-79262a3b0482","Type":"ContainerDied","Data":"7050af93cd46190b6402faa05f4eb4125521053cd5281ff4ae488466fcb3d276"} Oct 30 10:00:16.249806 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:16.249780 5938 scope.go:122] "RemoveContainer" containerID="96e8d1e73bb88da5d1180d2a832ca045cb41cfafec562823f65ab8c79bb7a17f" Oct 30 10:00:17.176500 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:17.176468 5938 generic.go:356] "Generic (PLEG): container finished" podID="067cbdb8-fb3e-422c-95ca-51d281a9dd76" containerID="f1d293d5ce5ed5022b83189f962556cf4f1048db8bebd6e84000d4e0616798b6" exitCode=60 Oct 30 10:00:17.176701 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:17.176546 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" event={"ID":"067cbdb8-fb3e-422c-95ca-51d281a9dd76","Type":"ContainerDied","Data":"f1d293d5ce5ed5022b83189f962556cf4f1048db8bebd6e84000d4e0616798b6"} Oct 30 10:00:17.176701 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:17.176636 5938 scope.go:122] "RemoveContainer" containerID="96e8d1e73bb88da5d1180d2a832ca045cb41cfafec562823f65ab8c79bb7a17f" Oct 30 10:00:17.177157 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:17.177140 5938 scope.go:122] "RemoveContainer" containerID="f1d293d5ce5ed5022b83189f962556cf4f1048db8bebd6e84000d4e0616798b6" Oct 30 10:00:17.177446 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:00:17.177424 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4_odh-ai-gateway-infra(067cbdb8-fb3e-422c-95ca-51d281a9dd76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" podUID="067cbdb8-fb3e-422c-95ca-51d281a9dd76" Oct 30 10:00:17.326613 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:17.326575 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" Oct 30 10:00:17.472891 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:17.472812 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9d4f8d1-4871-4e09-a254-79262a3b0482-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d4f8d1-4871-4e09-a254-79262a3b0482-maas-api-xks-ca\") pod \"f9d4f8d1-4871-4e09-a254-79262a3b0482\" (UID: \"f9d4f8d1-4871-4e09-a254-79262a3b0482\") " Oct 30 10:00:17.472891 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:17.472856 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d4f8d1-4871-4e09-a254-79262a3b0482-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d4f8d1-4871-4e09-a254-79262a3b0482-maas-api-openshift-ca\") pod \"f9d4f8d1-4871-4e09-a254-79262a3b0482\" (UID: \"f9d4f8d1-4871-4e09-a254-79262a3b0482\") " Oct 30 10:00:17.473100 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:17.473059 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d4f8d1-4871-4e09-a254-79262a3b0482-kube-api-access-9vk5f\" (UniqueName: \"kubernetes.io/projected/f9d4f8d1-4871-4e09-a254-79262a3b0482-kube-api-access-9vk5f\") pod \"f9d4f8d1-4871-4e09-a254-79262a3b0482\" (UID: \"f9d4f8d1-4871-4e09-a254-79262a3b0482\") " Oct 30 10:00:17.473259 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:17.473236 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d4f8d1-4871-4e09-a254-79262a3b0482-maas-api-openshift-ca" pod "f9d4f8d1-4871-4e09-a254-79262a3b0482" (UID: "f9d4f8d1-4871-4e09-a254-79262a3b0482"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:17.473394 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:17.473380 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d4f8d1-4871-4e09-a254-79262a3b0482-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:17.475145 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:17.475111 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d4f8d1-4871-4e09-a254-79262a3b0482-maas-api-xks-ca" pod "f9d4f8d1-4871-4e09-a254-79262a3b0482" (UID: "f9d4f8d1-4871-4e09-a254-79262a3b0482"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:00:17.475280 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:17.475255 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d4f8d1-4871-4e09-a254-79262a3b0482-kube-api-access-9vk5f" pod "f9d4f8d1-4871-4e09-a254-79262a3b0482" (UID: "f9d4f8d1-4871-4e09-a254-79262a3b0482"). InnerVolumeSpecName "kube-api-access-9vk5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:17.574674 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:17.574640 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vk5f\" (UniqueName: \"kubernetes.io/projected/f9d4f8d1-4871-4e09-a254-79262a3b0482-kube-api-access-9vk5f\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:17.574674 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:17.574669 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d4f8d1-4871-4e09-a254-79262a3b0482-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.090306 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.090279 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 10:00:18.179818 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.179793 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-podinfo\") pod \"f2b81448-0b6f-4093-9bdc-37139355b7e1\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " Oct 30 10:00:18.179818 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.179831 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-data\") pod \"f2b81448-0b6f-4093-9bdc-37139355b7e1\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " Oct 30 10:00:18.180093 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.179910 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-credential-socket\") pod \"f2b81448-0b6f-4093-9bdc-37139355b7e1\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " Oct 30 10:00:18.180093 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.179963 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-workload-certs\") pod \"f2b81448-0b6f-4093-9bdc-37139355b7e1\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " Oct 30 10:00:18.180093 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.180008 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b81448-0b6f-4093-9bdc-37139355b7e1-kube-api-access-qjzmv\" (UniqueName: \"kubernetes.io/projected/f2b81448-0b6f-4093-9bdc-37139355b7e1-kube-api-access-qjzmv\") pod \"f2b81448-0b6f-4093-9bdc-37139355b7e1\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " Oct 30 10:00:18.180093 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.180041 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-workload-socket\") pod \"f2b81448-0b6f-4093-9bdc-37139355b7e1\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " Oct 30 10:00:18.180093 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.180088 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-envoy\") pod \"f2b81448-0b6f-4093-9bdc-37139355b7e1\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " Oct 30 10:00:18.180443 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.180144 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-token\" (UniqueName: \"kubernetes.io/projected/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-token\") pod \"f2b81448-0b6f-4093-9bdc-37139355b7e1\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " Oct 30 10:00:18.180443 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.180178 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b81448-0b6f-4093-9bdc-37139355b7e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b81448-0b6f-4093-9bdc-37139355b7e1-istiod-ca-cert\") pod \"f2b81448-0b6f-4093-9bdc-37139355b7e1\" (UID: \"f2b81448-0b6f-4093-9bdc-37139355b7e1\") " Oct 30 10:00:18.180443 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.180224 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-data" pod "f2b81448-0b6f-4093-9bdc-37139355b7e1" (UID: "f2b81448-0b6f-4093-9bdc-37139355b7e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.180443 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.180335 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-workload-certs" pod "f2b81448-0b6f-4093-9bdc-37139355b7e1" (UID: "f2b81448-0b6f-4093-9bdc-37139355b7e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.180701 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.180457 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-workload-socket" pod "f2b81448-0b6f-4093-9bdc-37139355b7e1" (UID: "f2b81448-0b6f-4093-9bdc-37139355b7e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.180701 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.180659 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b81448-0b6f-4093-9bdc-37139355b7e1-istiod-ca-cert" pod "f2b81448-0b6f-4093-9bdc-37139355b7e1" (UID: "f2b81448-0b6f-4093-9bdc-37139355b7e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:18.180816 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.180710 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.180816 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.180730 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.180816 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.180743 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.180989 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.180955 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-credential-socket" pod "f2b81448-0b6f-4093-9bdc-37139355b7e1" (UID: "f2b81448-0b6f-4093-9bdc-37139355b7e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.183282 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.183255 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b81448-0b6f-4093-9bdc-37139355b7e1-kube-api-access-qjzmv" pod "f2b81448-0b6f-4093-9bdc-37139355b7e1" (UID: "f2b81448-0b6f-4093-9bdc-37139355b7e1"). InnerVolumeSpecName "kube-api-access-qjzmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:18.183282 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.183266 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-envoy" pod "f2b81448-0b6f-4093-9bdc-37139355b7e1" (UID: "f2b81448-0b6f-4093-9bdc-37139355b7e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.183613 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.183575 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-token" pod "f2b81448-0b6f-4093-9bdc-37139355b7e1" (UID: "f2b81448-0b6f-4093-9bdc-37139355b7e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:18.184698 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.184656 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-podinfo" pod "f2b81448-0b6f-4093-9bdc-37139355b7e1" (UID: "f2b81448-0b6f-4093-9bdc-37139355b7e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 10:00:18.190420 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.190391 5938 generic.go:356] "Generic (PLEG): container finished" podID="f2b81448-0b6f-4093-9bdc-37139355b7e1" containerID="3872aa546cdf36a12365fd99d37eb06528a6b8607721d89e8925211c8e5b60c9" exitCode=0 Oct 30 10:00:18.190765 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.190459 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" Oct 30 10:00:18.190765 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.190523 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" event={"ID":"f2b81448-0b6f-4093-9bdc-37139355b7e1","Type":"ContainerDied","Data":"3872aa546cdf36a12365fd99d37eb06528a6b8607721d89e8925211c8e5b60c9"} Oct 30 10:00:18.190765 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.190752 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt" event={"ID":"f2b81448-0b6f-4093-9bdc-37139355b7e1","Type":"ContainerDied","Data":"d89497ba28e195e731eb99e495ac66ff79716e841f4a8b504a6835c0f26df851"} Oct 30 10:00:18.190931 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.190795 5938 scope.go:122] "RemoveContainer" containerID="3872aa546cdf36a12365fd99d37eb06528a6b8607721d89e8925211c8e5b60c9" Oct 30 10:00:18.192705 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.192446 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" Oct 30 10:00:18.192705 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.192466 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp" event={"ID":"f9d4f8d1-4871-4e09-a254-79262a3b0482","Type":"ContainerDied","Data":"a6ad8d203743ea1e01c57549cc0f4a4950098010584a6602f618ff0cb28c0e5d"} Oct 30 10:00:18.192705 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.192507 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6ad8d203743ea1e01c57549cc0f4a4950098010584a6602f618ff0cb28c0e5d" Oct 30 10:00:18.211137 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.211120 5938 scope.go:122] "RemoveContainer" containerID="3872aa546cdf36a12365fd99d37eb06528a6b8607721d89e8925211c8e5b60c9" Oct 30 10:00:18.211424 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:00:18.211407 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3872aa546cdf36a12365fd99d37eb06528a6b8607721d89e8925211c8e5b60c9\": container with ID starting with 3872aa546cdf36a12365fd99d37eb06528a6b8607721d89e8925211c8e5b60c9 not found: ID does not exist" containerID="3872aa546cdf36a12365fd99d37eb06528a6b8607721d89e8925211c8e5b60c9" Oct 30 10:00:18.211481 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.211430 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3872aa546cdf36a12365fd99d37eb06528a6b8607721d89e8925211c8e5b60c9"} err="failed to get container status \"3872aa546cdf36a12365fd99d37eb06528a6b8607721d89e8925211c8e5b60c9\": rpc error: code = NotFound desc = could not find container \"3872aa546cdf36a12365fd99d37eb06528a6b8607721d89e8925211c8e5b60c9\": container with ID starting with 3872aa546cdf36a12365fd99d37eb06528a6b8607721d89e8925211c8e5b60c9 not found: ID does not exist" Oct 30 10:00:18.235916 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.235893 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4"] Oct 30 10:00:18.269490 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.269449 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt"] Oct 30 10:00:18.271376 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.271353 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-54cxt"] Oct 30 10:00:18.282234 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.282210 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.282340 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.282236 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.282340 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.282250 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b81448-0b6f-4093-9bdc-37139355b7e1-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.282340 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.282263 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b81448-0b6f-4093-9bdc-37139355b7e1-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.282340 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.282278 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b81448-0b6f-4093-9bdc-37139355b7e1-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.282340 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.282293 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjzmv\" (UniqueName: \"kubernetes.io/projected/f2b81448-0b6f-4093-9bdc-37139355b7e1-kube-api-access-qjzmv\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.356229 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.356201 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" Oct 30 10:00:18.484125 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.484057 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/067cbdb8-fb3e-422c-95ca-51d281a9dd76-kube-api-access-x2xlz\" (UniqueName: \"kubernetes.io/projected/067cbdb8-fb3e-422c-95ca-51d281a9dd76-kube-api-access-x2xlz\") pod \"067cbdb8-fb3e-422c-95ca-51d281a9dd76\" (UID: \"067cbdb8-fb3e-422c-95ca-51d281a9dd76\") " Oct 30 10:00:18.484125 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.484108 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/067cbdb8-fb3e-422c-95ca-51d281a9dd76-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/067cbdb8-fb3e-422c-95ca-51d281a9dd76-maas-api-service-ca\") pod \"067cbdb8-fb3e-422c-95ca-51d281a9dd76\" (UID: \"067cbdb8-fb3e-422c-95ca-51d281a9dd76\") " Oct 30 10:00:18.484521 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.484489 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067cbdb8-fb3e-422c-95ca-51d281a9dd76-maas-api-service-ca" pod "067cbdb8-fb3e-422c-95ca-51d281a9dd76" (UID: "067cbdb8-fb3e-422c-95ca-51d281a9dd76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:18.486404 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.486385 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067cbdb8-fb3e-422c-95ca-51d281a9dd76-kube-api-access-x2xlz" pod "067cbdb8-fb3e-422c-95ca-51d281a9dd76" (UID: "067cbdb8-fb3e-422c-95ca-51d281a9dd76"). InnerVolumeSpecName "kube-api-access-x2xlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:18.585470 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.585446 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2xlz\" (UniqueName: \"kubernetes.io/projected/067cbdb8-fb3e-422c-95ca-51d281a9dd76-kube-api-access-x2xlz\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.585470 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:18.585467 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/067cbdb8-fb3e-422c-95ca-51d281a9dd76-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:19.201174 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:19.201082 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" event={"ID":"067cbdb8-fb3e-422c-95ca-51d281a9dd76","Type":"ContainerDied","Data":"d37779b7a816f8c255bd6cc53ee39bcf5560fd89bc853522ddb72865d0c2aab4"} Oct 30 10:00:19.201174 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:19.201111 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4" Oct 30 10:00:19.201174 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:19.201138 5938 scope.go:122] "RemoveContainer" containerID="f1d293d5ce5ed5022b83189f962556cf4f1048db8bebd6e84000d4e0616798b6" Oct 30 10:00:19.255207 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:19.255179 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4"] Oct 30 10:00:19.262359 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:19.262329 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889240-njkl4"] Oct 30 10:00:20.255988 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:20.255926 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="067cbdb8-fb3e-422c-95ca-51d281a9dd76" path="/var/lib/kubelet/pods/067cbdb8-fb3e-422c-95ca-51d281a9dd76/volumes" Oct 30 10:00:20.256470 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:20.256429 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b81448-0b6f-4093-9bdc-37139355b7e1" path="/var/lib/kubelet/pods/f2b81448-0b6f-4093-9bdc-37139355b7e1/volumes" Oct 30 10:00:24.678110 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:24.678077 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg"] Oct 30 10:00:29.678823 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:29.678761 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" podUID="bc95bf70-8335-4690-aea8-86934dd0f73f" containerName="payload-pre-processing" containerID="cri-o://05746cc8740517ee77db67100a307c520056c2f3f0b4434a1051ef0f337d26a4" gracePeriod=25 Oct 30 10:00:30.381808 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:30.381771 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp"] Oct 30 10:00:30.395299 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:30.395268 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f58f32-bnjhp"] Oct 30 10:00:32.256639 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:32.256578 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d4f8d1-4871-4e09-a254-79262a3b0482" path="/var/lib/kubelet/pods/f9d4f8d1-4871-4e09-a254-79262a3b0482/volumes" Oct 30 10:00:37.356856 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.356822 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" podUID="bc95bf70-8335-4690-aea8-86934dd0f73f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 10:00:37.758788 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.758756 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb"] Oct 30 10:00:37.759150 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.759138 5938 cpu_manager.go:414] "containerMap: removing container" podUID="067cbdb8-fb3e-422c-95ca-51d281a9dd76" containerName="cleanup" Oct 30 10:00:37.759197 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.759152 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="067cbdb8-fb3e-422c-95ca-51d281a9dd76" containerName="cleanup" Oct 30 10:00:37.759197 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.759170 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f9d4f8d1-4871-4e09-a254-79262a3b0482" containerName="revoke-keys" Oct 30 10:00:37.759197 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.759175 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d4f8d1-4871-4e09-a254-79262a3b0482" containerName="revoke-keys" Oct 30 10:00:37.759197 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.759194 5938 cpu_manager.go:414] "containerMap: removing container" podUID="067cbdb8-fb3e-422c-95ca-51d281a9dd76" containerName="cleanup" Oct 30 10:00:37.759329 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.759199 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="067cbdb8-fb3e-422c-95ca-51d281a9dd76" containerName="cleanup" Oct 30 10:00:37.759329 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.759215 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f2b81448-0b6f-4093-9bdc-37139355b7e1" containerName="istio-proxy" Oct 30 10:00:37.759329 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.759219 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b81448-0b6f-4093-9bdc-37139355b7e1" containerName="istio-proxy" Oct 30 10:00:37.759329 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.759279 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d4f8d1-4871-4e09-a254-79262a3b0482" containerName="revoke-keys" Oct 30 10:00:37.759329 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.759289 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="067cbdb8-fb3e-422c-95ca-51d281a9dd76" containerName="cleanup" Oct 30 10:00:37.759329 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.759297 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="067cbdb8-fb3e-422c-95ca-51d281a9dd76" containerName="cleanup" Oct 30 10:00:37.759329 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.759303 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b81448-0b6f-4093-9bdc-37139355b7e1" containerName="istio-proxy" Oct 30 10:00:37.759329 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.759310 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="067cbdb8-fb3e-422c-95ca-51d281a9dd76" containerName="cleanup" Oct 30 10:00:37.763946 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.763926 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb" Oct 30 10:00:37.781469 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.781434 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb"] Oct 30 10:00:37.792877 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.792852 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 10:00:37.792999 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.792887 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 10:00:37.861954 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.861925 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3353409-273d-4b86-9641-ad408e74fdcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb\" (UID: \"f3353409-273d-4b86-9641-ad408e74fdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb" Oct 30 10:00:37.862076 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.861962 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3353409-273d-4b86-9641-ad408e74fdcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb\" (UID: \"f3353409-273d-4b86-9641-ad408e74fdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb" Oct 30 10:00:37.862135 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.862117 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7thkp\" (UniqueName: \"kubernetes.io/projected/f3353409-273d-4b86-9641-ad408e74fdcf-kube-api-access-7thkp\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb\" (UID: \"f3353409-273d-4b86-9641-ad408e74fdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb" Oct 30 10:00:37.962510 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.962484 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7thkp\" (UniqueName: \"kubernetes.io/projected/f3353409-273d-4b86-9641-ad408e74fdcf-kube-api-access-7thkp\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb\" (UID: \"f3353409-273d-4b86-9641-ad408e74fdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb" Oct 30 10:00:37.962706 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.962530 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3353409-273d-4b86-9641-ad408e74fdcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb\" (UID: \"f3353409-273d-4b86-9641-ad408e74fdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb" Oct 30 10:00:37.962706 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.962555 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3353409-273d-4b86-9641-ad408e74fdcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb\" (UID: \"f3353409-273d-4b86-9641-ad408e74fdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb" Oct 30 10:00:37.963363 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.963331 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3353409-273d-4b86-9641-ad408e74fdcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb\" (UID: \"f3353409-273d-4b86-9641-ad408e74fdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb" Oct 30 10:00:37.965416 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.965391 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3353409-273d-4b86-9641-ad408e74fdcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb\" (UID: \"f3353409-273d-4b86-9641-ad408e74fdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb" Oct 30 10:00:37.983158 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:37.983129 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7thkp\" (UniqueName: \"kubernetes.io/projected/f3353409-273d-4b86-9641-ad408e74fdcf-kube-api-access-7thkp\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb\" (UID: \"f3353409-273d-4b86-9641-ad408e74fdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb" Oct 30 10:00:38.077417 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:38.077367 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb" Oct 30 10:00:38.374249 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:38.374223 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb"] Oct 30 10:00:38.376324 ip-10-0-135-148 kubenswrapper[5938]: W1030 10:00:38.376296 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3353409_273d_4b86_9641_ad408e74fdcf.slice/crio-9e6d24aee20c6a6585e831077738eb50ece6fbd00a7a0ea0ef296d2b50714d67 WatchSource:0}: Error finding container 9e6d24aee20c6a6585e831077738eb50ece6fbd00a7a0ea0ef296d2b50714d67: Status 404 returned error can't find the container with id 9e6d24aee20c6a6585e831077738eb50ece6fbd00a7a0ea0ef296d2b50714d67 Oct 30 10:00:39.330004 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:39.329968 5938 generic.go:356] "Generic (PLEG): container finished" podID="f3353409-273d-4b86-9641-ad408e74fdcf" containerID="135cf43f1957c63ad25261e9c13206ca0f2dce1cf104e82a81eb4352738ac12d" exitCode=0 Oct 30 10:00:39.330181 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:39.330035 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb" event={"ID":"f3353409-273d-4b86-9641-ad408e74fdcf","Type":"ContainerDied","Data":"135cf43f1957c63ad25261e9c13206ca0f2dce1cf104e82a81eb4352738ac12d"} Oct 30 10:00:39.330181 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:39.330072 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb" event={"ID":"f3353409-273d-4b86-9641-ad408e74fdcf","Type":"ContainerStarted","Data":"9e6d24aee20c6a6585e831077738eb50ece6fbd00a7a0ea0ef296d2b50714d67"} Oct 30 10:00:40.489433 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:40.489411 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb" Oct 30 10:00:40.593832 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:40.593764 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3353409-273d-4b86-9641-ad408e74fdcf-kube-api-access-7thkp\" (UniqueName: \"kubernetes.io/projected/f3353409-273d-4b86-9641-ad408e74fdcf-kube-api-access-7thkp\") pod \"f3353409-273d-4b86-9641-ad408e74fdcf\" (UID: \"f3353409-273d-4b86-9641-ad408e74fdcf\") " Oct 30 10:00:40.593832 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:40.593828 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3353409-273d-4b86-9641-ad408e74fdcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3353409-273d-4b86-9641-ad408e74fdcf-maas-api-xks-ca\") pod \"f3353409-273d-4b86-9641-ad408e74fdcf\" (UID: \"f3353409-273d-4b86-9641-ad408e74fdcf\") " Oct 30 10:00:40.594065 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:40.593851 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3353409-273d-4b86-9641-ad408e74fdcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3353409-273d-4b86-9641-ad408e74fdcf-maas-api-openshift-ca\") pod \"f3353409-273d-4b86-9641-ad408e74fdcf\" (UID: \"f3353409-273d-4b86-9641-ad408e74fdcf\") " Oct 30 10:00:40.594245 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:40.594219 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3353409-273d-4b86-9641-ad408e74fdcf-maas-api-openshift-ca" pod "f3353409-273d-4b86-9641-ad408e74fdcf" (UID: "f3353409-273d-4b86-9641-ad408e74fdcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:40.596076 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:40.596052 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3353409-273d-4b86-9641-ad408e74fdcf-maas-api-xks-ca" pod "f3353409-273d-4b86-9641-ad408e74fdcf" (UID: "f3353409-273d-4b86-9641-ad408e74fdcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:00:40.596216 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:40.596124 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3353409-273d-4b86-9641-ad408e74fdcf-kube-api-access-7thkp" pod "f3353409-273d-4b86-9641-ad408e74fdcf" (UID: "f3353409-273d-4b86-9641-ad408e74fdcf"). InnerVolumeSpecName "kube-api-access-7thkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:40.694859 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:40.694834 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7thkp\" (UniqueName: \"kubernetes.io/projected/f3353409-273d-4b86-9641-ad408e74fdcf-kube-api-access-7thkp\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:40.694859 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:40.694855 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3353409-273d-4b86-9641-ad408e74fdcf-maas-api-xks-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:40.695023 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:40.694864 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3353409-273d-4b86-9641-ad408e74fdcf-maas-api-openshift-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:41.343266 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:41.343235 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb" Oct 30 10:00:41.343440 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:41.343262 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb" event={"ID":"f3353409-273d-4b86-9641-ad408e74fdcf","Type":"ContainerDied","Data":"9e6d24aee20c6a6585e831077738eb50ece6fbd00a7a0ea0ef296d2b50714d67"} Oct 30 10:00:41.343440 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:41.343317 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e6d24aee20c6a6585e831077738eb50ece6fbd00a7a0ea0ef296d2b50714d67" Oct 30 10:00:45.040657 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:45.040624 5938 scope.go:122] "RemoveContainer" containerID="36b1b9add788e5cc7528151351f064ec8f6141b31c6e511b8c0673e9a52947bf" Oct 30 10:00:45.050139 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:45.050110 5938 scope.go:122] "RemoveContainer" containerID="e0e6335b8ef181b09fd7f0814f16f5af559164e601c4412957f54b915f41c308" Oct 30 10:00:45.058515 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:45.058495 5938 scope.go:122] "RemoveContainer" containerID="942c1e2b9492029c5b4e286026b3d2efd33db2eab049ddcd9d96f0c8b808b134" Oct 30 10:00:45.807678 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:00:45.807640 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 10:00:47.357138 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:47.357097 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" podUID="bc95bf70-8335-4690-aea8-86934dd0f73f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 10:00:49.928372 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:49.928347 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" Oct 30 10:00:49.979817 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:49.979750 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc95bf70-8335-4690-aea8-86934dd0f73f-kube-api-access-ptb6q\" (UniqueName: \"kubernetes.io/projected/bc95bf70-8335-4690-aea8-86934dd0f73f-kube-api-access-ptb6q\") pod \"bc95bf70-8335-4690-aea8-86934dd0f73f\" (UID: \"bc95bf70-8335-4690-aea8-86934dd0f73f\") " Oct 30 10:00:49.979817 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:49.979804 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc95bf70-8335-4690-aea8-86934dd0f73f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc95bf70-8335-4690-aea8-86934dd0f73f-plugins-config-volume\") pod \"bc95bf70-8335-4690-aea8-86934dd0f73f\" (UID: \"bc95bf70-8335-4690-aea8-86934dd0f73f\") " Oct 30 10:00:49.980277 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:49.980251 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc95bf70-8335-4690-aea8-86934dd0f73f-plugins-config-volume" pod "bc95bf70-8335-4690-aea8-86934dd0f73f" (UID: "bc95bf70-8335-4690-aea8-86934dd0f73f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:49.982060 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:49.982039 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc95bf70-8335-4690-aea8-86934dd0f73f-kube-api-access-ptb6q" pod "bc95bf70-8335-4690-aea8-86934dd0f73f" (UID: "bc95bf70-8335-4690-aea8-86934dd0f73f"). InnerVolumeSpecName "kube-api-access-ptb6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:50.081126 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:50.081097 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptb6q\" (UniqueName: \"kubernetes.io/projected/bc95bf70-8335-4690-aea8-86934dd0f73f-kube-api-access-ptb6q\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:50.081126 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:50.081123 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc95bf70-8335-4690-aea8-86934dd0f73f-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:00:50.399445 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:50.399403 5938 generic.go:356] "Generic (PLEG): container finished" podID="bc95bf70-8335-4690-aea8-86934dd0f73f" containerID="05746cc8740517ee77db67100a307c520056c2f3f0b4434a1051ef0f337d26a4" exitCode=0 Oct 30 10:00:50.399656 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:50.399436 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" event={"ID":"bc95bf70-8335-4690-aea8-86934dd0f73f","Type":"ContainerDied","Data":"05746cc8740517ee77db67100a307c520056c2f3f0b4434a1051ef0f337d26a4"} Oct 30 10:00:50.399656 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:50.399506 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" event={"ID":"bc95bf70-8335-4690-aea8-86934dd0f73f","Type":"ContainerDied","Data":"5bf4569a47ce1aa15d2d9b2b9607bcf4dcb1a57237f1d597e28c6ac1df458707"} Oct 30 10:00:50.399656 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:50.399447 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg" Oct 30 10:00:50.399656 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:50.399536 5938 scope.go:122] "RemoveContainer" containerID="05746cc8740517ee77db67100a307c520056c2f3f0b4434a1051ef0f337d26a4" Oct 30 10:00:50.410062 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:50.409664 5938 scope.go:122] "RemoveContainer" containerID="05746cc8740517ee77db67100a307c520056c2f3f0b4434a1051ef0f337d26a4" Oct 30 10:00:50.410422 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:00:50.410310 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05746cc8740517ee77db67100a307c520056c2f3f0b4434a1051ef0f337d26a4\": container with ID starting with 05746cc8740517ee77db67100a307c520056c2f3f0b4434a1051ef0f337d26a4 not found: ID does not exist" containerID="05746cc8740517ee77db67100a307c520056c2f3f0b4434a1051ef0f337d26a4" Oct 30 10:00:50.410422 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:50.410353 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05746cc8740517ee77db67100a307c520056c2f3f0b4434a1051ef0f337d26a4"} err="failed to get container status \"05746cc8740517ee77db67100a307c520056c2f3f0b4434a1051ef0f337d26a4\": rpc error: code = NotFound desc = could not find container \"05746cc8740517ee77db67100a307c520056c2f3f0b4434a1051ef0f337d26a4\": container with ID starting with 05746cc8740517ee77db67100a307c520056c2f3f0b4434a1051ef0f337d26a4 not found: ID does not exist" Oct 30 10:00:50.453653 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:50.453629 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg"] Oct 30 10:00:50.466870 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:50.466835 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f58f32-6d98748cbc-gj5qg"] Oct 30 10:00:52.256815 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:52.256740 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc95bf70-8335-4690-aea8-86934dd0f73f" path="/var/lib/kubelet/pods/bc95bf70-8335-4690-aea8-86934dd0f73f/volumes" Oct 30 10:00:54.560133 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:54.560097 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb"] Oct 30 10:00:54.573477 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:54.573448 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ebd65-f8qkb"] Oct 30 10:00:56.255894 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:00:56.255858 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3353409-273d-4b86-9641-ad408e74fdcf" path="/var/lib/kubelet/pods/f3353409-273d-4b86-9641-ad408e74fdcf/volumes" Oct 30 10:01:15.807749 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:01:15.807707 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 10:01:26.798671 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:26.798637 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh"] Oct 30 10:01:26.812331 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:26.812306 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl"] Oct 30 10:01:41.819967 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:41.819928 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" containerID="cri-o://33a8a76f360a750f0f09a68b5271a41dd9f04914f405c274eab81e313410a901" gracePeriod=45 Oct 30 10:01:41.841025 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:41.840968 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" containerID="cri-o://dfbff12f5fe892fd3cf5a6e5e855ee130e7bab065cb0575801b4fd8ef1b2d99f" gracePeriod=45 Oct 30 10:01:42.032996 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.032959 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk"] Oct 30 10:01:42.046742 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.046712 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c9c63e-hpjdk"] Oct 30 10:01:42.080455 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.080400 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 10:01:42.108274 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.108242 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 10:01:42.146181 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.146131 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-model-cache\") pod \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " Oct 30 10:01:42.146181 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.146179 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-tmp-dir\") pod \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " Oct 30 10:01:42.146457 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.146219 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-home\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-home\") pod \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " Oct 30 10:01:42.146457 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.146290 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-model-cache\") pod \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " Oct 30 10:01:42.146457 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.146345 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-tmp-dir\") pod \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " Oct 30 10:01:42.146457 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.146389 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89552945-90ed-4c3c-b93a-a9e106aa4bf8-kube-api-access-7722f\" (UniqueName: \"kubernetes.io/projected/89552945-90ed-4c3c-b93a-a9e106aa4bf8-kube-api-access-7722f\") pod \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " Oct 30 10:01:42.146457 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.146424 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-dshm\") pod \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " Oct 30 10:01:42.146799 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.146468 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89552945-90ed-4c3c-b93a-a9e106aa4bf8-tls-certs\" (UniqueName: \"kubernetes.io/secret/89552945-90ed-4c3c-b93a-a9e106aa4bf8-tls-certs\") pod \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " Oct 30 10:01:42.146799 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.146507 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-home" pod "89552945-90ed-4c3c-b93a-a9e106aa4bf8" (UID: "89552945-90ed-4c3c-b93a-a9e106aa4bf8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:42.146799 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.146546 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-kube-api-access-q5nwm\" (UniqueName: \"kubernetes.io/projected/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-kube-api-access-q5nwm\") pod \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " Oct 30 10:01:42.146799 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.146554 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-model-cache" pod "89552945-90ed-4c3c-b93a-a9e106aa4bf8" (UID: "89552945-90ed-4c3c-b93a-a9e106aa4bf8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:42.146799 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.146581 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-tls-certs\") pod \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " Oct 30 10:01:42.146799 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.146637 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-dshm\") pod \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\" (UID: \"89552945-90ed-4c3c-b93a-a9e106aa4bf8\") " Oct 30 10:01:42.146799 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.146690 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-home\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-home\") pod \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\" (UID: \"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb\") " Oct 30 10:01:42.147078 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.146865 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-model-cache" pod "15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" (UID: "15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:42.147153 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.147130 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-home" pod "15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" (UID: "15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:42.147343 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.147309 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-tmp-dir" pod "15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" (UID: "15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:42.147460 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.147350 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.147460 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.147373 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.147460 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.147386 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.147673 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.147606 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-tmp-dir" pod "89552945-90ed-4c3c-b93a-a9e106aa4bf8" (UID: "89552945-90ed-4c3c-b93a-a9e106aa4bf8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:42.149744 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.149719 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-kube-api-access-q5nwm" pod "15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" (UID: "15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb"). InnerVolumeSpecName "kube-api-access-q5nwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:42.149744 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.149733 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-dshm" pod "15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" (UID: "15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:42.149924 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.149766 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89552945-90ed-4c3c-b93a-a9e106aa4bf8-tls-certs" pod "89552945-90ed-4c3c-b93a-a9e106aa4bf8" (UID: "89552945-90ed-4c3c-b93a-a9e106aa4bf8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:01:42.149924 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.149879 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-dshm" pod "89552945-90ed-4c3c-b93a-a9e106aa4bf8" (UID: "89552945-90ed-4c3c-b93a-a9e106aa4bf8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:42.150213 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.150188 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89552945-90ed-4c3c-b93a-a9e106aa4bf8-kube-api-access-7722f" pod "89552945-90ed-4c3c-b93a-a9e106aa4bf8" (UID: "89552945-90ed-4c3c-b93a-a9e106aa4bf8"). InnerVolumeSpecName "kube-api-access-7722f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:42.151691 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.151660 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-tls-certs" pod "15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" (UID: "15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:01:42.247895 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.247866 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.247895 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.247890 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.247895 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.247899 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89552945-90ed-4c3c-b93a-a9e106aa4bf8-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.248133 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.247907 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.248133 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.247917 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7722f\" (UniqueName: \"kubernetes.io/projected/89552945-90ed-4c3c-b93a-a9e106aa4bf8-kube-api-access-7722f\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.248133 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.247925 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.248133 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.247933 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89552945-90ed-4c3c-b93a-a9e106aa4bf8-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.248133 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.247941 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5nwm\" (UniqueName: \"kubernetes.io/projected/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-kube-api-access-q5nwm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.248133 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.247948 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.258908 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.258870 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e40a8f23-0eaa-4785-940d-595544bbf8df" path="/var/lib/kubelet/pods/e40a8f23-0eaa-4785-940d-595544bbf8df/volumes" Oct 30 10:01:42.706423 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.706392 5938 generic.go:356] "Generic (PLEG): container finished" podID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerID="dfbff12f5fe892fd3cf5a6e5e855ee130e7bab065cb0575801b4fd8ef1b2d99f" exitCode=0 Oct 30 10:01:42.706575 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.706444 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" Oct 30 10:01:42.706575 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.706476 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" event={"ID":"89552945-90ed-4c3c-b93a-a9e106aa4bf8","Type":"ContainerDied","Data":"dfbff12f5fe892fd3cf5a6e5e855ee130e7bab065cb0575801b4fd8ef1b2d99f"} Oct 30 10:01:42.706575 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.706521 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl" event={"ID":"89552945-90ed-4c3c-b93a-a9e106aa4bf8","Type":"ContainerDied","Data":"77589f64c9670e36df339afa410b92c9c4db2a7a8e47fbf683c8a6278edbf3bc"} Oct 30 10:01:42.706575 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.706543 5938 scope.go:122] "RemoveContainer" containerID="dfbff12f5fe892fd3cf5a6e5e855ee130e7bab065cb0575801b4fd8ef1b2d99f" Oct 30 10:01:42.708094 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.708073 5938 generic.go:356] "Generic (PLEG): container finished" podID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerID="33a8a76f360a750f0f09a68b5271a41dd9f04914f405c274eab81e313410a901" exitCode=0 Oct 30 10:01:42.708207 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.708121 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" event={"ID":"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb","Type":"ContainerDied","Data":"33a8a76f360a750f0f09a68b5271a41dd9f04914f405c274eab81e313410a901"} Oct 30 10:01:42.708207 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.708149 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" event={"ID":"15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb","Type":"ContainerDied","Data":"fb0099b738223af1496e69c19a05a4116fd81ca5f8d803cadbc0cd990250052e"} Oct 30 10:01:42.708318 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.708202 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh" Oct 30 10:01:42.715839 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.715822 5938 scope.go:122] "RemoveContainer" containerID="dfbff12f5fe892fd3cf5a6e5e855ee130e7bab065cb0575801b4fd8ef1b2d99f" Oct 30 10:01:42.716133 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:01:42.716115 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfbff12f5fe892fd3cf5a6e5e855ee130e7bab065cb0575801b4fd8ef1b2d99f\": container with ID starting with dfbff12f5fe892fd3cf5a6e5e855ee130e7bab065cb0575801b4fd8ef1b2d99f not found: ID does not exist" containerID="dfbff12f5fe892fd3cf5a6e5e855ee130e7bab065cb0575801b4fd8ef1b2d99f" Oct 30 10:01:42.716233 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.716141 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfbff12f5fe892fd3cf5a6e5e855ee130e7bab065cb0575801b4fd8ef1b2d99f"} err="failed to get container status \"dfbff12f5fe892fd3cf5a6e5e855ee130e7bab065cb0575801b4fd8ef1b2d99f\": rpc error: code = NotFound desc = could not find container \"dfbff12f5fe892fd3cf5a6e5e855ee130e7bab065cb0575801b4fd8ef1b2d99f\": container with ID starting with dfbff12f5fe892fd3cf5a6e5e855ee130e7bab065cb0575801b4fd8ef1b2d99f not found: ID does not exist" Oct 30 10:01:42.716233 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.716163 5938 scope.go:122] "RemoveContainer" containerID="33a8a76f360a750f0f09a68b5271a41dd9f04914f405c274eab81e313410a901" Oct 30 10:01:42.724490 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.724474 5938 scope.go:122] "RemoveContainer" containerID="33a8a76f360a750f0f09a68b5271a41dd9f04914f405c274eab81e313410a901" Oct 30 10:01:42.724782 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:01:42.724764 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a8a76f360a750f0f09a68b5271a41dd9f04914f405c274eab81e313410a901\": container with ID starting with 33a8a76f360a750f0f09a68b5271a41dd9f04914f405c274eab81e313410a901 not found: ID does not exist" containerID="33a8a76f360a750f0f09a68b5271a41dd9f04914f405c274eab81e313410a901" Oct 30 10:01:42.724835 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.724788 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a8a76f360a750f0f09a68b5271a41dd9f04914f405c274eab81e313410a901"} err="failed to get container status \"33a8a76f360a750f0f09a68b5271a41dd9f04914f405c274eab81e313410a901\": rpc error: code = NotFound desc = could not find container \"33a8a76f360a750f0f09a68b5271a41dd9f04914f405c274eab81e313410a901\": container with ID starting with 33a8a76f360a750f0f09a68b5271a41dd9f04914f405c274eab81e313410a901 not found: ID does not exist" Oct 30 10:01:42.759704 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.759683 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl"] Oct 30 10:01:42.772809 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.772786 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/simulator-w0-2f69c8-kserve-9d8c74b8f-wdjbl"] Oct 30 10:01:42.798652 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.798631 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh"] Oct 30 10:01:42.800459 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:42.800442 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/premium-w0-2f69c8-kserve-b987d7c4d-jc9xh"] Oct 30 10:01:44.255629 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:44.255601 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" path="/var/lib/kubelet/pods/15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb/volumes" Oct 30 10:01:44.256036 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:44.256004 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" path="/var/lib/kubelet/pods/89552945-90ed-4c3c-b93a-a9e106aa4bf8/volumes" Oct 30 10:01:45.152582 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:45.152551 5938 scope.go:122] "RemoveContainer" containerID="15b077cf3d86627a2bf34e632e4d0f143d3a897cc56ee83bd21539f8b917fdf1" Oct 30 10:01:45.161505 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:45.161487 5938 scope.go:122] "RemoveContainer" containerID="1e8c877b99f3e980e841d177506d2711a4362c27fb8872306f9e39450c1bd9bd" Oct 30 10:01:45.169621 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:45.169581 5938 scope.go:122] "RemoveContainer" containerID="ec5e45b831b1964f12417aca53695e4c38f0602941721e949f20a69b59a31d31" Oct 30 10:01:45.177737 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:45.177718 5938 scope.go:122] "RemoveContainer" containerID="0dc8ee77f71b8929cf385ca6260784eaee8872e21edfd6821f9cabfbb4de2319" Oct 30 10:01:45.185868 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:45.185850 5938 scope.go:122] "RemoveContainer" containerID="b0531c41bc03a1695c6fcfec33132bd285c6d9c2bfd96e071d0e009395db0eca" Oct 30 10:01:45.808103 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:01:45.808057 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 10:01:53.024812 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:53.024771 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn"] Oct 30 10:01:53.038171 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:53.038136 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c2c1da1-6z5nn"] Oct 30 10:01:54.256512 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:54.256477 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6892707d-6965-4b58-8e66-283364988a03" path="/var/lib/kubelet/pods/6892707d-6965-4b58-8e66-283364988a03/volumes" Oct 30 10:01:54.635055 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:54.635016 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6"] Oct 30 10:01:54.635300 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:54.635249 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" podUID="fddb9f28-1159-49f5-8982-7e0c8275df4a" containerName="istio-proxy" containerID="cri-o://20087ed558a52d49cc2cdaa63fa58637332462035248de52de79c6d3a9c934bb" gracePeriod=30 Oct 30 10:01:56.027828 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:56.027795 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5"] Oct 30 10:01:56.041694 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:56.041671 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q"] Oct 30 10:01:56.042958 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:56.042938 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7766142-lg8t5"] Oct 30 10:01:56.044084 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:56.044060 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a710ab-hqk2q"] Oct 30 10:01:56.255108 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:56.255067 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d68d8a-8f84-4419-8344-cfe9d883178e" path="/var/lib/kubelet/pods/20d68d8a-8f84-4419-8344-cfe9d883178e/volumes" Oct 30 10:01:56.255568 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:56.255547 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22161d0f-3e3c-44db-acfb-81ac4f26a55c" path="/var/lib/kubelet/pods/22161d0f-3e3c-44db-acfb-81ac4f26a55c/volumes" Oct 30 10:01:59.198869 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:59.198841 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-f64pg_f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1/manager/0.log" Oct 30 10:01:59.814707 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:59.814680 5938 generic.go:356] "Generic (PLEG): container finished" podID="fddb9f28-1159-49f5-8982-7e0c8275df4a" containerID="20087ed558a52d49cc2cdaa63fa58637332462035248de52de79c6d3a9c934bb" exitCode=0 Oct 30 10:01:59.814863 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:59.814753 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" event={"ID":"fddb9f28-1159-49f5-8982-7e0c8275df4a","Type":"ContainerDied","Data":"20087ed558a52d49cc2cdaa63fa58637332462035248de52de79c6d3a9c934bb"} Oct 30 10:01:59.894969 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:01:59.894949 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 10:02:00.002376 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.002307 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-workload-certs\") pod \"fddb9f28-1159-49f5-8982-7e0c8275df4a\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " Oct 30 10:02:00.002376 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.002354 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-podinfo\") pod \"fddb9f28-1159-49f5-8982-7e0c8275df4a\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " Oct 30 10:02:00.002618 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.002388 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-data\") pod \"fddb9f28-1159-49f5-8982-7e0c8275df4a\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " Oct 30 10:02:00.002618 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.002431 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-token\" (UniqueName: \"kubernetes.io/projected/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-token\") pod \"fddb9f28-1159-49f5-8982-7e0c8275df4a\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " Oct 30 10:02:00.002618 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.002515 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-credential-socket\") pod \"fddb9f28-1159-49f5-8982-7e0c8275df4a\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " Oct 30 10:02:00.002618 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.002543 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-envoy\") pod \"fddb9f28-1159-49f5-8982-7e0c8275df4a\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " Oct 30 10:02:00.002618 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.002609 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-workload-socket\") pod \"fddb9f28-1159-49f5-8982-7e0c8275df4a\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " Oct 30 10:02:00.002913 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.002636 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fddb9f28-1159-49f5-8982-7e0c8275df4a-kube-api-access-thrtx\" (UniqueName: \"kubernetes.io/projected/fddb9f28-1159-49f5-8982-7e0c8275df4a-kube-api-access-thrtx\") pod \"fddb9f28-1159-49f5-8982-7e0c8275df4a\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " Oct 30 10:02:00.002913 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.002641 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-workload-certs" pod "fddb9f28-1159-49f5-8982-7e0c8275df4a" (UID: "fddb9f28-1159-49f5-8982-7e0c8275df4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:02:00.002913 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.002667 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fddb9f28-1159-49f5-8982-7e0c8275df4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fddb9f28-1159-49f5-8982-7e0c8275df4a-istiod-ca-cert\") pod \"fddb9f28-1159-49f5-8982-7e0c8275df4a\" (UID: \"fddb9f28-1159-49f5-8982-7e0c8275df4a\") " Oct 30 10:02:00.002913 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.002865 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-credential-socket" pod "fddb9f28-1159-49f5-8982-7e0c8275df4a" (UID: "fddb9f28-1159-49f5-8982-7e0c8275df4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:02:00.002913 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.002881 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-workload-socket" pod "fddb9f28-1159-49f5-8982-7e0c8275df4a" (UID: "fddb9f28-1159-49f5-8982-7e0c8275df4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:02:00.003179 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.003004 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-data" pod "fddb9f28-1159-49f5-8982-7e0c8275df4a" (UID: "fddb9f28-1159-49f5-8982-7e0c8275df4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:02:00.003179 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.003086 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:02:00.003179 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.003108 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:02:00.003179 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.003123 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:02:00.003353 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.003178 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fddb9f28-1159-49f5-8982-7e0c8275df4a-istiod-ca-cert" pod "fddb9f28-1159-49f5-8982-7e0c8275df4a" (UID: "fddb9f28-1159-49f5-8982-7e0c8275df4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:02:00.004789 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.004761 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-envoy" pod "fddb9f28-1159-49f5-8982-7e0c8275df4a" (UID: "fddb9f28-1159-49f5-8982-7e0c8275df4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:02:00.005071 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.005051 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fddb9f28-1159-49f5-8982-7e0c8275df4a-kube-api-access-thrtx" pod "fddb9f28-1159-49f5-8982-7e0c8275df4a" (UID: "fddb9f28-1159-49f5-8982-7e0c8275df4a"). InnerVolumeSpecName "kube-api-access-thrtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:02:00.005201 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.005186 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-podinfo" pod "fddb9f28-1159-49f5-8982-7e0c8275df4a" (UID: "fddb9f28-1159-49f5-8982-7e0c8275df4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 10:02:00.005352 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.005330 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-token" pod "fddb9f28-1159-49f5-8982-7e0c8275df4a" (UID: "fddb9f28-1159-49f5-8982-7e0c8275df4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:02:00.037362 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.037343 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dkbbc_e57e45d7-2bbb-433c-aa34-e6f23c81efe1/manager/0.log" Oct 30 10:02:00.103696 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.103676 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:02:00.103696 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.103695 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:02:00.103845 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.103705 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:02:00.103845 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.103712 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fddb9f28-1159-49f5-8982-7e0c8275df4a-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:02:00.103845 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.103720 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thrtx\" (UniqueName: \"kubernetes.io/projected/fddb9f28-1159-49f5-8982-7e0c8275df4a-kube-api-access-thrtx\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:02:00.103845 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.103730 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fddb9f28-1159-49f5-8982-7e0c8275df4a-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\"" Oct 30 10:02:00.821691 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.821662 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" Oct 30 10:02:00.821691 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.821688 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6" event={"ID":"fddb9f28-1159-49f5-8982-7e0c8275df4a","Type":"ContainerDied","Data":"851f1b74e6258a661fc266ec9a4593f56262c04c2e3341123ce54806bf298921"} Oct 30 10:02:00.822186 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.821750 5938 scope.go:122] "RemoveContainer" containerID="20087ed558a52d49cc2cdaa63fa58637332462035248de52de79c6d3a9c934bb" Oct 30 10:02:00.890024 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.889999 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6"] Oct 30 10:02:00.893844 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:00.893822 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5b7d6"] Oct 30 10:02:02.255864 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:02.255834 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fddb9f28-1159-49f5-8982-7e0c8275df4a" path="/var/lib/kubelet/pods/fddb9f28-1159-49f5-8982-7e0c8275df4a/volumes" Oct 30 10:02:12.613660 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:12.613623 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-f64pg_f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1/manager/0.log" Oct 30 10:02:13.457982 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:13.457951 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dkbbc_e57e45d7-2bbb-433c-aa34-e6f23c81efe1/manager/0.log" Oct 30 10:02:14.464015 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:14.463983 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5d696658ff-242p9_8e16e34a-4935-43de-b1db-af983af6fc0b/maas-api/0.log" Oct 30 10:02:14.891317 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:14.891296 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-284af7d5-kngd4_3f56fdd1-d21a-4007-bf18-1d76cb519e65/revoke-keys/0.log" Oct 30 10:02:15.707166 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:15.707133 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9e9fb720-n8cbb_85510896-49d4-4b91-b576-444dd1dba1fe/revoke-keys/0.log" Oct 30 10:02:15.807362 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:02:15.807328 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 10:02:16.139712 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:16.139690 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c50a5045-gj7vt_8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef/revoke-keys/0.log" Oct 30 10:02:16.528271 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:16.528196 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-eb40e476-zhn6v_9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50/revoke-keys/0.log" Oct 30 10:02:19.025821 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:19.025784 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb"] Oct 30 10:02:19.039418 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:19.039391 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v"] Oct 30 10:02:19.040508 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:19.040493 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb40e476-zhn6v"] Oct 30 10:02:19.041535 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:19.041513 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9fb720-n8cbb"] Oct 30 10:02:20.255001 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:20.254964 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85510896-49d4-4b91-b576-444dd1dba1fe" path="/var/lib/kubelet/pods/85510896-49d4-4b91-b576-444dd1dba1fe/volumes" Oct 30 10:02:20.255423 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:20.255408 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50" path="/var/lib/kubelet/pods/9a9dc48b-8fdb-4ec1-ac2e-c447a2a96a50/volumes" Oct 30 10:02:21.559206 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:21.559176 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-767d5cbf5-8tqvw_12b2c3e7-3797-46e8-85df-93d84f58d580/kube-auth-proxy/0.log" Oct 30 10:02:22.362271 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:22.362202 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-k5n95_a2a093a8-bad3-4c10-b148-357f42f7910c/istio-proxy/0.log" Oct 30 10:02:22.578891 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:22.578867 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-vnxfr_f38877f2-7419-459b-a30c-8a55b146cb48/istio-proxy/0.log" Oct 30 10:02:32.027460 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:32.027426 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt"] Oct 30 10:02:32.040802 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:32.040778 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c50a5045-gj7vt"] Oct 30 10:02:32.256426 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:32.256395 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef" path="/var/lib/kubelet/pods/8e8e41b9-4e7b-4df9-9a2c-80289cbe4cef/volumes" Oct 30 10:02:36.356855 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.356822 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lt4lw/must-gather-24c4b"] Oct 30 10:02:36.357259 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357191 5938 cpu_manager.go:414] "containerMap: removing container" podUID="067cbdb8-fb3e-422c-95ca-51d281a9dd76" containerName="cleanup" Oct 30 10:02:36.357259 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357203 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="067cbdb8-fb3e-422c-95ca-51d281a9dd76" containerName="cleanup" Oct 30 10:02:36.357259 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357222 5938 cpu_manager.go:414] "containerMap: removing container" podUID="fddb9f28-1159-49f5-8982-7e0c8275df4a" containerName="istio-proxy" Oct 30 10:02:36.357259 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357227 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fddb9f28-1159-49f5-8982-7e0c8275df4a" containerName="istio-proxy" Oct 30 10:02:36.357259 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357242 5938 cpu_manager.go:414] "containerMap: removing container" podUID="bc95bf70-8335-4690-aea8-86934dd0f73f" containerName="payload-pre-processing" Oct 30 10:02:36.357259 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357247 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc95bf70-8335-4690-aea8-86934dd0f73f" containerName="payload-pre-processing" Oct 30 10:02:36.357259 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357261 5938 cpu_manager.go:414] "containerMap: removing container" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" Oct 30 10:02:36.357531 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357266 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" Oct 30 10:02:36.357531 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357278 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f3353409-273d-4b86-9641-ad408e74fdcf" containerName="revoke-keys" Oct 30 10:02:36.357531 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357282 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3353409-273d-4b86-9641-ad408e74fdcf" containerName="revoke-keys" Oct 30 10:02:36.357531 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357292 5938 cpu_manager.go:414] "containerMap: removing container" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" Oct 30 10:02:36.357531 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357297 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" Oct 30 10:02:36.357531 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357351 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="fddb9f28-1159-49f5-8982-7e0c8275df4a" containerName="istio-proxy" Oct 30 10:02:36.357531 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357361 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc95bf70-8335-4690-aea8-86934dd0f73f" containerName="payload-pre-processing" Oct 30 10:02:36.357531 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357368 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="89552945-90ed-4c3c-b93a-a9e106aa4bf8" containerName="main" Oct 30 10:02:36.357531 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357375 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3353409-273d-4b86-9641-ad408e74fdcf" containerName="revoke-keys" Oct 30 10:02:36.357531 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.357381 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="15fcfc84-b3ad-4d47-93a9-1ab16e9f79fb" containerName="main" Oct 30 10:02:36.361457 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.361434 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lt4lw/must-gather-24c4b" Oct 30 10:02:36.394107 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.394084 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lt4lw\"/\"openshift-service-ca.crt\"" Oct 30 10:02:36.394285 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.394152 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lt4lw\"/\"kube-root-ca.crt\"" Oct 30 10:02:36.394285 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.394239 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lt4lw\"/\"default-dockercfg-9t8kn\"" Oct 30 10:02:36.395384 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.395366 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lt4lw/must-gather-24c4b"] Oct 30 10:02:36.407524 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.407503 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s45tr\" (UniqueName: \"kubernetes.io/projected/00f11001-dca9-4287-8772-94dcd64bef07-kube-api-access-s45tr\") pod \"must-gather-24c4b\" (UID: \"00f11001-dca9-4287-8772-94dcd64bef07\") " pod="openshift-must-gather-lt4lw/must-gather-24c4b" Oct 30 10:02:36.407659 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.407536 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/00f11001-dca9-4287-8772-94dcd64bef07-must-gather-output\") pod \"must-gather-24c4b\" (UID: \"00f11001-dca9-4287-8772-94dcd64bef07\") " pod="openshift-must-gather-lt4lw/must-gather-24c4b" Oct 30 10:02:36.455973 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.455951 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qd5lv/must-gather-p6wjq"] Oct 30 10:02:36.459560 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.459546 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qd5lv/must-gather-p6wjq" Oct 30 10:02:36.479611 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.479561 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qd5lv/must-gather-p6wjq"] Oct 30 10:02:36.480257 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.480237 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qd5lv\"/\"openshift-service-ca.crt\"" Oct 30 10:02:36.480344 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.480273 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qd5lv\"/\"kube-root-ca.crt\"" Oct 30 10:02:36.480615 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.480579 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qd5lv\"/\"default-dockercfg-57j6b\"" Oct 30 10:02:36.508496 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.508472 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9hpx\" (UniqueName: \"kubernetes.io/projected/dc7e8632-7f94-49f1-ab62-5efc373dae1e-kube-api-access-x9hpx\") pod \"must-gather-p6wjq\" (UID: \"dc7e8632-7f94-49f1-ab62-5efc373dae1e\") " pod="openshift-must-gather-qd5lv/must-gather-p6wjq" Oct 30 10:02:36.508639 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.508523 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s45tr\" (UniqueName: \"kubernetes.io/projected/00f11001-dca9-4287-8772-94dcd64bef07-kube-api-access-s45tr\") pod \"must-gather-24c4b\" (UID: \"00f11001-dca9-4287-8772-94dcd64bef07\") " pod="openshift-must-gather-lt4lw/must-gather-24c4b" Oct 30 10:02:36.508639 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.508546 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/00f11001-dca9-4287-8772-94dcd64bef07-must-gather-output\") pod \"must-gather-24c4b\" (UID: \"00f11001-dca9-4287-8772-94dcd64bef07\") " pod="openshift-must-gather-lt4lw/must-gather-24c4b" Oct 30 10:02:36.508776 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.508739 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dc7e8632-7f94-49f1-ab62-5efc373dae1e-must-gather-output\") pod \"must-gather-p6wjq\" (UID: \"dc7e8632-7f94-49f1-ab62-5efc373dae1e\") " pod="openshift-must-gather-qd5lv/must-gather-p6wjq" Oct 30 10:02:36.508987 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.508969 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/00f11001-dca9-4287-8772-94dcd64bef07-must-gather-output\") pod \"must-gather-24c4b\" (UID: \"00f11001-dca9-4287-8772-94dcd64bef07\") " pod="openshift-must-gather-lt4lw/must-gather-24c4b" Oct 30 10:02:36.529080 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.529065 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s45tr\" (UniqueName: \"kubernetes.io/projected/00f11001-dca9-4287-8772-94dcd64bef07-kube-api-access-s45tr\") pod \"must-gather-24c4b\" (UID: \"00f11001-dca9-4287-8772-94dcd64bef07\") " pod="openshift-must-gather-lt4lw/must-gather-24c4b" Oct 30 10:02:36.609664 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.609575 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dc7e8632-7f94-49f1-ab62-5efc373dae1e-must-gather-output\") pod \"must-gather-p6wjq\" (UID: \"dc7e8632-7f94-49f1-ab62-5efc373dae1e\") " pod="openshift-must-gather-qd5lv/must-gather-p6wjq" Oct 30 10:02:36.609664 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.609641 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9hpx\" (UniqueName: \"kubernetes.io/projected/dc7e8632-7f94-49f1-ab62-5efc373dae1e-kube-api-access-x9hpx\") pod \"must-gather-p6wjq\" (UID: \"dc7e8632-7f94-49f1-ab62-5efc373dae1e\") " pod="openshift-must-gather-qd5lv/must-gather-p6wjq" Oct 30 10:02:36.609932 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.609916 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dc7e8632-7f94-49f1-ab62-5efc373dae1e-must-gather-output\") pod \"must-gather-p6wjq\" (UID: \"dc7e8632-7f94-49f1-ab62-5efc373dae1e\") " pod="openshift-must-gather-qd5lv/must-gather-p6wjq" Oct 30 10:02:36.630561 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.630542 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9hpx\" (UniqueName: \"kubernetes.io/projected/dc7e8632-7f94-49f1-ab62-5efc373dae1e-kube-api-access-x9hpx\") pod \"must-gather-p6wjq\" (UID: \"dc7e8632-7f94-49f1-ab62-5efc373dae1e\") " pod="openshift-must-gather-qd5lv/must-gather-p6wjq" Oct 30 10:02:36.676516 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.676495 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lt4lw/must-gather-24c4b" Oct 30 10:02:36.773944 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.773912 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qd5lv/must-gather-p6wjq" Oct 30 10:02:36.971082 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.971054 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lt4lw/must-gather-24c4b"] Oct 30 10:02:36.974246 ip-10-0-135-148 kubenswrapper[5938]: W1030 10:02:36.974222 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00f11001_dca9_4287_8772_94dcd64bef07.slice/crio-43a5ebf51486c734ba6a8a7b5e70910fd5e066a1a858f910b4895e395f354783 WatchSource:0}: Error finding container 43a5ebf51486c734ba6a8a7b5e70910fd5e066a1a858f910b4895e395f354783: Status 404 returned error can't find the container with id 43a5ebf51486c734ba6a8a7b5e70910fd5e066a1a858f910b4895e395f354783 Oct 30 10:02:36.976011 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:36.975991 5938 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 10:02:37.014482 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:37.014453 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lt4lw/must-gather-24c4b" event={"ID":"00f11001-dca9-4287-8772-94dcd64bef07","Type":"ContainerStarted","Data":"43a5ebf51486c734ba6a8a7b5e70910fd5e066a1a858f910b4895e395f354783"} Oct 30 10:02:37.070417 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:37.070396 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qd5lv/must-gather-p6wjq"] Oct 30 10:02:37.072858 ip-10-0-135-148 kubenswrapper[5938]: W1030 10:02:37.072833 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc7e8632_7f94_49f1_ab62_5efc373dae1e.slice/crio-b39f9bdcc00eb3cf8c9c61c4ea5a83e69a6330140760bb29974dd5caa6f18fdf WatchSource:0}: Error finding container b39f9bdcc00eb3cf8c9c61c4ea5a83e69a6330140760bb29974dd5caa6f18fdf: Status 404 returned error can't find the container with id b39f9bdcc00eb3cf8c9c61c4ea5a83e69a6330140760bb29974dd5caa6f18fdf Oct 30 10:02:38.027010 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:38.026934 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qd5lv/must-gather-p6wjq" event={"ID":"dc7e8632-7f94-49f1-ab62-5efc373dae1e","Type":"ContainerStarted","Data":"b39f9bdcc00eb3cf8c9c61c4ea5a83e69a6330140760bb29974dd5caa6f18fdf"} Oct 30 10:02:42.905429 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:42.905393 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m5sz6_c7400965-54de-4420-a62a-37baf38d949d/global-pull-secret-syncer/0.log" Oct 30 10:02:44.077775 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:44.077678 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qd5lv/must-gather-p6wjq" event={"ID":"dc7e8632-7f94-49f1-ab62-5efc373dae1e","Type":"ContainerStarted","Data":"36bde156a81491507a9964a607ed0db891fe662eff7ddaa21ad8b7f03311550b"} Oct 30 10:02:44.077775 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:44.077743 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qd5lv/must-gather-p6wjq" event={"ID":"dc7e8632-7f94-49f1-ab62-5efc373dae1e","Type":"ContainerStarted","Data":"0d4d995ea6139cef207c59701212e0dabc7c7cae2c4e673d77a9616be42aa72e"} Oct 30 10:02:44.079708 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:44.079677 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lt4lw/must-gather-24c4b" event={"ID":"00f11001-dca9-4287-8772-94dcd64bef07","Type":"ContainerStarted","Data":"716e416323e06aa321ebfd74fd37d7f139d755f982c87f538d0d25bcf495b4cd"} Oct 30 10:02:44.079900 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:44.079718 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lt4lw/must-gather-24c4b" event={"ID":"00f11001-dca9-4287-8772-94dcd64bef07","Type":"ContainerStarted","Data":"17172fe579433a7b9af30e549f8147b5f4a1cf19fb8ee72d9e08ce498aed738f"} Oct 30 10:02:44.117614 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:44.117435 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qd5lv/must-gather-p6wjq" podStartSLOduration=1.59511603 podStartE2EDuration="8.117413953s" totalImagesPullingTime="6.522297923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:02:44.116815626 +0000 UTC m=+4081.979521447" watchObservedRunningTime="2026-10-30 10:02:44.117413953 +0000 UTC m=+4081.980119775" Oct 30 10:02:44.161088 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:44.161040 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lt4lw/must-gather-24c4b" podStartSLOduration=1.616385894 podStartE2EDuration="8.161026608s" totalImagesPullingTime="6.544640714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:02:44.158309518 +0000 UTC m=+4082.021015340" watchObservedRunningTime="2026-10-30 10:02:44.161026608 +0000 UTC m=+4082.023732429" Oct 30 10:02:44.164607 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:44.164556 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hsqk7_8b4cd17b-9e6f-41d3-970f-9c19d3d1e13b/konnectivity-agent/0.log" Oct 30 10:02:44.393058 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:44.393010 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-148.ec2.internal_9307784112acd4d13d34c64b57f18e71/haproxy/0.log" Oct 30 10:02:45.232887 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:45.232858 5938 scope.go:122] "RemoveContainer" containerID="0a4a35d4824db862ecdea0cd5f7d3014ff78994fd3666794b160f01a52e7bcc8" Oct 30 10:02:45.244926 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:45.244757 5938 scope.go:122] "RemoveContainer" containerID="d62422f910c2329ff74db8ba6903f920ff99dd2027724028d5bffb4573993fb0" Oct 30 10:02:45.255520 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:45.255502 5938 scope.go:122] "RemoveContainer" containerID="f411c1e0dfc029ca4dc30bad38d0000aca0f1974687e26886c70e1b6b1a7ee62" Oct 30 10:02:45.264325 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:45.264306 5938 scope.go:122] "RemoveContainer" containerID="a8ad664f851c58b428a67bc445ec79db6f63b5e76ffe4a59d70cc5d799d1a82b" Oct 30 10:02:45.274352 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:45.274334 5938 scope.go:122] "RemoveContainer" containerID="b77ee8109f829e99882656d427bf63a8e1956461074907c6ac61c7b556c46f13" Oct 30 10:02:45.808765 ip-10-0-135-148 kubenswrapper[5938]: E1030 10:02:45.808729 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vbz6m" containerName="extractor" Oct 30 10:02:47.298486 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:47.298451 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-f64pg_f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1/manager/0.log" Oct 30 10:02:47.340056 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:47.340025 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-f64pg_f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1/manager/0.log" Oct 30 10:02:51.769170 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:51.769136 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dkbbc_e57e45d7-2bbb-433c-aa34-e6f23c81efe1/manager/0.log" Oct 30 10:02:51.802884 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:51.802857 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dkbbc_e57e45d7-2bbb-433c-aa34-e6f23c81efe1/manager/0.log" Oct 30 10:02:52.991351 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:52.991320 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf"] Oct 30 10:02:52.998623 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:52.998250 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.014035 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.014008 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf"] Oct 30 10:02:53.026282 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.026259 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6llqt\"/\"kube-root-ca.crt\"" Oct 30 10:02:53.026282 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.026274 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6llqt\"/\"openshift-service-ca.crt\"" Oct 30 10:02:53.026463 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.026299 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6llqt\"/\"default-dockercfg-tfjr2\"" Oct 30 10:02:53.160968 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.160930 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d313ae7-5917-4282-97f2-20ab433fa4ab-proc\") pod \"perf-node-gather-daemonset-7kxqf\" (UID: \"2d313ae7-5917-4282-97f2-20ab433fa4ab\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.161172 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.160984 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d313ae7-5917-4282-97f2-20ab433fa4ab-sys\") pod \"perf-node-gather-daemonset-7kxqf\" (UID: \"2d313ae7-5917-4282-97f2-20ab433fa4ab\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.161172 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.161059 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d313ae7-5917-4282-97f2-20ab433fa4ab-podres\") pod \"perf-node-gather-daemonset-7kxqf\" (UID: \"2d313ae7-5917-4282-97f2-20ab433fa4ab\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.161172 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.161102 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d313ae7-5917-4282-97f2-20ab433fa4ab-lib-modules\") pod \"perf-node-gather-daemonset-7kxqf\" (UID: \"2d313ae7-5917-4282-97f2-20ab433fa4ab\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.161350 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.161210 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjpwz\" (UniqueName: \"kubernetes.io/projected/2d313ae7-5917-4282-97f2-20ab433fa4ab-kube-api-access-gjpwz\") pod \"perf-node-gather-daemonset-7kxqf\" (UID: \"2d313ae7-5917-4282-97f2-20ab433fa4ab\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.261913 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.261844 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d313ae7-5917-4282-97f2-20ab433fa4ab-proc\") pod \"perf-node-gather-daemonset-7kxqf\" (UID: \"2d313ae7-5917-4282-97f2-20ab433fa4ab\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.261913 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.261890 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d313ae7-5917-4282-97f2-20ab433fa4ab-sys\") pod \"perf-node-gather-daemonset-7kxqf\" (UID: \"2d313ae7-5917-4282-97f2-20ab433fa4ab\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.261913 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.261908 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d313ae7-5917-4282-97f2-20ab433fa4ab-podres\") pod \"perf-node-gather-daemonset-7kxqf\" (UID: \"2d313ae7-5917-4282-97f2-20ab433fa4ab\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.262131 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.261928 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d313ae7-5917-4282-97f2-20ab433fa4ab-lib-modules\") pod \"perf-node-gather-daemonset-7kxqf\" (UID: \"2d313ae7-5917-4282-97f2-20ab433fa4ab\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.262131 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.261978 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjpwz\" (UniqueName: \"kubernetes.io/projected/2d313ae7-5917-4282-97f2-20ab433fa4ab-kube-api-access-gjpwz\") pod \"perf-node-gather-daemonset-7kxqf\" (UID: \"2d313ae7-5917-4282-97f2-20ab433fa4ab\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.262131 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.261981 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d313ae7-5917-4282-97f2-20ab433fa4ab-sys\") pod \"perf-node-gather-daemonset-7kxqf\" (UID: \"2d313ae7-5917-4282-97f2-20ab433fa4ab\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.262131 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.261983 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d313ae7-5917-4282-97f2-20ab433fa4ab-podres\") pod \"perf-node-gather-daemonset-7kxqf\" (UID: \"2d313ae7-5917-4282-97f2-20ab433fa4ab\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.262131 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.261982 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d313ae7-5917-4282-97f2-20ab433fa4ab-proc\") pod \"perf-node-gather-daemonset-7kxqf\" (UID: \"2d313ae7-5917-4282-97f2-20ab433fa4ab\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.262131 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.262032 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d313ae7-5917-4282-97f2-20ab433fa4ab-lib-modules\") pod \"perf-node-gather-daemonset-7kxqf\" (UID: \"2d313ae7-5917-4282-97f2-20ab433fa4ab\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.282486 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.282463 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjpwz\" (UniqueName: \"kubernetes.io/projected/2d313ae7-5917-4282-97f2-20ab433fa4ab-kube-api-access-gjpwz\") pod \"perf-node-gather-daemonset-7kxqf\" (UID: \"2d313ae7-5917-4282-97f2-20ab433fa4ab\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.311287 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.311263 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:53.618548 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:53.618518 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf"] Oct 30 10:02:53.626934 ip-10-0-135-148 kubenswrapper[5938]: W1030 10:02:53.626901 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2d313ae7_5917_4282_97f2_20ab433fa4ab.slice/crio-a80a71d7d455ec100610fee93243dd61b00021b992a10df41d955f1033c49788 WatchSource:0}: Error finding container a80a71d7d455ec100610fee93243dd61b00021b992a10df41d955f1033c49788: Status 404 returned error can't find the container with id a80a71d7d455ec100610fee93243dd61b00021b992a10df41d955f1033c49788 Oct 30 10:02:54.147962 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:54.147922 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" event={"ID":"2d313ae7-5917-4282-97f2-20ab433fa4ab","Type":"ContainerStarted","Data":"2a4746e078d5443fde28856c622018da428449a69a6a6d134c3047acffac150a"} Oct 30 10:02:54.148694 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:54.148024 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" event={"ID":"2d313ae7-5917-4282-97f2-20ab433fa4ab","Type":"ContainerStarted","Data":"a80a71d7d455ec100610fee93243dd61b00021b992a10df41d955f1033c49788"} Oct 30 10:02:54.148694 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:54.148063 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:02:54.189833 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:54.189780 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" podStartSLOduration=2.189766251 podStartE2EDuration="2.189766251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 10:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:02:54.188611109 +0000 UTC m=+4092.051316930" watchObservedRunningTime="2026-10-30 10:02:54.189766251 +0000 UTC m=+4092.052472071" Oct 30 10:02:54.943443 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:54.943416 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4eea09ac-fa40-47ea-affb-0e54be762c78/alertmanager/0.log" Oct 30 10:02:54.964376 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:54.964343 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-f64pg_f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1/manager/0.log" Oct 30 10:02:55.042017 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:55.041982 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-f64pg_f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1/manager/0.log" Oct 30 10:02:55.064255 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:55.064222 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4eea09ac-fa40-47ea-affb-0e54be762c78/config-reloader/0.log" Oct 30 10:02:55.186978 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:55.186951 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4eea09ac-fa40-47ea-affb-0e54be762c78/kube-rbac-proxy-web/0.log" Oct 30 10:02:55.308453 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:55.308342 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4eea09ac-fa40-47ea-affb-0e54be762c78/kube-rbac-proxy/0.log" Oct 30 10:02:55.431060 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:55.431031 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4eea09ac-fa40-47ea-affb-0e54be762c78/kube-rbac-proxy-metric/0.log" Oct 30 10:02:55.552425 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:55.552384 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4eea09ac-fa40-47ea-affb-0e54be762c78/prom-label-proxy/0.log" Oct 30 10:02:55.675044 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:55.675017 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4eea09ac-fa40-47ea-affb-0e54be762c78/init-config-reloader/0.log" Oct 30 10:02:55.944917 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:55.944844 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-9tlbg_1ecaa283-05a2-4f68-857e-dd14c857c804/kube-state-metrics/0.log" Oct 30 10:02:56.047036 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:56.047004 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-9tlbg_1ecaa283-05a2-4f68-857e-dd14c857c804/kube-rbac-proxy-main/0.log" Oct 30 10:02:56.171082 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:56.171046 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-9tlbg_1ecaa283-05a2-4f68-857e-dd14c857c804/kube-rbac-proxy-self/0.log" Oct 30 10:02:56.539947 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:56.539911 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8kgvx_6a763f73-62ad-43ed-babf-fb2ced67616a/node-exporter/0.log" Oct 30 10:02:56.658381 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:56.658346 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8kgvx_6a763f73-62ad-43ed-babf-fb2ced67616a/kube-rbac-proxy/0.log" Oct 30 10:02:56.777509 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:56.777477 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8kgvx_6a763f73-62ad-43ed-babf-fb2ced67616a/init-textfile/0.log" Oct 30 10:02:57.623785 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:57.623718 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-cddd2_866d9622-0c68-4e07-94ef-94cabbd34f86/kube-rbac-proxy-main/0.log" Oct 30 10:02:57.744645 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:57.744603 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-cddd2_866d9622-0c68-4e07-94ef-94cabbd34f86/kube-rbac-proxy-self/0.log" Oct 30 10:02:57.864984 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:57.864909 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-cddd2_866d9622-0c68-4e07-94ef-94cabbd34f86/openshift-state-metrics/0.log" Oct 30 10:02:57.999928 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:57.999826 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d315846-1abf-45e2-80bd-07a0c1f2ac24/prometheus/0.log" Oct 30 10:02:58.116721 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:58.116689 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d315846-1abf-45e2-80bd-07a0c1f2ac24/config-reloader/0.log" Oct 30 10:02:58.236699 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:58.236628 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d315846-1abf-45e2-80bd-07a0c1f2ac24/thanos-sidecar/0.log" Oct 30 10:02:58.367409 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:58.367242 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d315846-1abf-45e2-80bd-07a0c1f2ac24/kube-rbac-proxy-web/0.log" Oct 30 10:02:58.482244 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:58.482187 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d315846-1abf-45e2-80bd-07a0c1f2ac24/kube-rbac-proxy/0.log" Oct 30 10:02:58.605400 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:58.605198 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d315846-1abf-45e2-80bd-07a0c1f2ac24/kube-rbac-proxy-thanos/0.log" Oct 30 10:02:58.724643 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:58.724507 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d315846-1abf-45e2-80bd-07a0c1f2ac24/init-config-reloader/0.log" Oct 30 10:02:58.870343 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:58.870313 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-z42cw_e23f66da-2587-4ce8-bb3a-a344e5dad910/prometheus-operator/0.log" Oct 30 10:02:58.970809 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:58.970753 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-z42cw_e23f66da-2587-4ce8-bb3a-a344e5dad910/kube-rbac-proxy/0.log" Oct 30 10:02:59.033702 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:59.033629 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dkbbc_e57e45d7-2bbb-433c-aa34-e6f23c81efe1/manager/0.log" Oct 30 10:02:59.111134 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:59.111105 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dkbbc_e57e45d7-2bbb-433c-aa34-e6f23c81efe1/manager/0.log" Oct 30 10:02:59.223881 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:59.223854 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65c85d64c4-d8zwz_3e6de072-eb99-48f0-ade6-f7fad4be241b/telemeter-client/0.log" Oct 30 10:02:59.345424 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:59.345359 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65c85d64c4-d8zwz_3e6de072-eb99-48f0-ade6-f7fad4be241b/reload/0.log" Oct 30 10:02:59.462475 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:59.462442 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65c85d64c4-d8zwz_3e6de072-eb99-48f0-ade6-f7fad4be241b/kube-rbac-proxy/0.log" Oct 30 10:02:59.586881 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:59.586853 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f48c4fbf4-lhblx_7c5d7c66-ce15-4d20-bc37-4d832f308d9f/thanos-query/0.log" Oct 30 10:02:59.749525 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:59.749499 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f48c4fbf4-lhblx_7c5d7c66-ce15-4d20-bc37-4d832f308d9f/kube-rbac-proxy-web/0.log" Oct 30 10:02:59.870919 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:59.870656 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f48c4fbf4-lhblx_7c5d7c66-ce15-4d20-bc37-4d832f308d9f/kube-rbac-proxy/0.log" Oct 30 10:02:59.991104 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:02:59.991075 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f48c4fbf4-lhblx_7c5d7c66-ce15-4d20-bc37-4d832f308d9f/prom-label-proxy/0.log" Oct 30 10:03:00.113616 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:03:00.111777 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f48c4fbf4-lhblx_7c5d7c66-ce15-4d20-bc37-4d832f308d9f/kube-rbac-proxy-rules/0.log" Oct 30 10:03:00.167574 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:03:00.167547 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-7kxqf" Oct 30 10:03:00.232920 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:03:00.232886 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f48c4fbf4-lhblx_7c5d7c66-ce15-4d20-bc37-4d832f308d9f/kube-rbac-proxy-metrics/0.log" Oct 30 10:03:02.094542 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:03:02.094508 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-f64pg_f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1/manager/0.log" Oct 30 10:03:02.217334 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:03:02.217302 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-f64pg_f7fb5ced-7f2c-4dfb-91d4-8fef493cf1d1/manager/0.log" Oct 30 10:03:06.301718 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:03:06.301681 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dkbbc_e57e45d7-2bbb-433c-aa34-e6f23c81efe1/manager/0.log" Oct 30 10:03:06.425471 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:03:06.425437 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dkbbc_e57e45d7-2bbb-433c-aa34-e6f23c81efe1/manager/0.log" Oct 30 10:03:07.835052 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:03:07.835020 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fxq45_f76551a8-8b0b-459d-86f5-84cf822e83b8/dns/0.log" Oct 30 10:03:07.954082 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:03:07.954050 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fxq45_f76551a8-8b0b-459d-86f5-84cf822e83b8/kube-rbac-proxy/0.log" Oct 30 10:03:08.566747 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:03:08.566717 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q8z8s_b3b0e2a3-516f-4bf7-ba8f-a4dec22e4eb6/dns-node-resolver/0.log" Oct 30 10:03:10.096185 ip-10-0-135-148 kubenswrapper[5938]: I1030 10:03:10.096152 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mnq7z_c032d4ff-047f-43a9-a27e-41affb60615d/node-ca/0.log"